============================================================================== couchbase logs (indexer.log) cbbrowse_logs indexer.log ============================================================================== Initializing write barrier = 8000 2021-08-16T13:47:02.076+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:47:02.076+00:00 [Info] Setting ipv6=false 2021-08-16T13:47:02.081+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:47:02.090+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:47:02.090+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:47:02.090+00:00 [Info] Setting log level to Info 2021-08-16T13:47:02.091+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021-08-16T13:47:02.092+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021/08/16 13:47:02 audit: created new audit service 2021-08-16T13:47:02.100+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:47:02.100+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:47:02.100+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:47:02.100+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:47:02.100+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:47:02.109+00:00 [Info] security context initialized 2021-08-16T13:47:02.109+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:47:02.109+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:47:02.109+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:47:02.109+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:47:02.109+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:47:02.116+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:47:02.117+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:47:02.119+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:47:02.131+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:47:02.132+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:47:02.136+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:47:02.138+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:47:02.138+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:47:02.139+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:47:02.141+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:47:02.165+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:47:02.165+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 62 us 2021-08-16T13:47:02.166+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:47:02.167+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:47:02.174+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:47:02.174+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:47:02.174+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:47:02.174+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:47:02.174+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:47:02.174+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:47:02.176+00:00 [Info] ClustMgr:handleSetLocalValue Key IndexerId Value 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:02.177+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:02.177+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:02.179+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err Post "http://ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999/adminport/shutdownTopicRequest": dial tcp 172.31.29.9:9999: connect: connection refused 2021-08-16T13:47:02.179+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received Post "http://ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999/adminport/shutdownTopicRequest": dial tcp 172.31.29.9:9999: connect: connection refused from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:47:03.185+00:00 [Error] KVSender::closeMutationStream, MAINT_STREAM Error from Projector Post "http://ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999/adminport/shutdownTopicRequest": dial tcp 172.31.29.9:9999: connect: connection refused 2021-08-16T13:47:03.185+00:00 [Warn] Indexer::closeAllStreams Stream MAINT_STREAM Projector health check needed, indexer can not proceed, Error received Post "http://ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999/adminport/shutdownTopicRequest": dial tcp 172.31.29.9:9999: connect: connection refused. Retrying (1). 2021-08-16T13:47:08.185+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:08.187+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:47:08.187+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:47:08.187+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:47:08.187+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:08.187+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:47:08.187+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:47:08.187+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:47:08.187+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:47:08.187+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:47:08.187+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:47:08.187+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:47:08.190+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap , elapsed: 2.31542ms 2021-08-16T13:47:08.190+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:47:08.192+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:47:08.192+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:08.192+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:47:08.192+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:47:08.192+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:47:08.194+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:47:08.198+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:47:08.198+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:47:08.198+00:00 [Warn] Encountered error while reading persisted stats. Skipping read. Error: open /opt/couchbase/var/lib/couchbase/data/@2i/indexstats/stats: no such file or directory 2021-08-16T13:47:08.198+00:00 [Info] ClustMgr:handleIndexerReady mType: CLUST_MGR_INDEXER_READY indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:47:08.198+00:00 [Info] SettingsMgr::handleIndexerReady apply pending setting changes 2021-08-16T13:47:08.198+00:00 [Info] RebalanceMgr::initService Init 2021-08-16T13:47:08.198+00:00 [Error] ServiceMgr::getGlobalTopology Error gathering global topology 127.0.0.1:9102/getIndexMetadata Unable to find given hostport in cbauth database: `127.0.0.1:9102' 2021-08-16T13:47:08.198+00:00 [Error] ServiceMgr::updateNodeList Error Fetching Topology Unable to find given hostport in cbauth database: `127.0.0.1:9102' 2021-08-16T13:47:08.198+00:00 [Info] ServiceMgr::registerWithServer nodeuuid 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:08.199+00:00 [Info] ClustMgr:handleSetLocalValue Key IndexerNodeUUID Value 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:08.199+00:00 [Info] ServiceMgr::listenMoveIndex &{8a95cde683482bdec453edbb8b18eb5d 5 } 2021-08-16T13:47:08.200+00:00 [Info] Indexer::NewIndexer Status Active 2021-08-16T13:47:08.200+00:00 [Info] 127.0.0.1:8091 starting RESTful services 2021-08-16T13:47:08.200+00:00 [Info] Indexer::monitorMemUsage started... 2021-08-16T13:47:08.200+00:00 [Info] memstats {"Alloc":0, "TotalAlloc":0, "Sys":0, "Lookups":0, "Mallocs":0,"Frees":0, "HeapAlloc":0, "HeapSys":0, "HeapIdle":0, "HeapInuse":0,"HeapReleased":0, "HeapObjects":0,"MSpanInuse":0, "MSpanSys": 0, "StackInuse": 0,"GCSys":0, "LastGC":0,"PauseTotalNs":0, "PauseNs":[], "NumGC":0} 2021-08-16T13:47:08.200+00:00 [Info] progress stats collection done. 2021-08-16T13:47:08.200+00:00 [Info] send progress stats to clients 2021-08-16T13:47:08.201+00:00 [Info] New settings received: {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false} 2021-08-16T13:47:08.202+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:47:08.202+00:00 [Info] Setting log level to Info 2021-08-16T13:47:08.204+00:00 [Info] ServiceMgr::GetTaskList [] 2021-08-16T13:47:08.204+00:00 [Info] ServiceMgr::GetTaskList returns &{[0 0 0 0 0 0 0 0] []} 2021-08-16T13:47:08.205+00:00 [Info] ServiceMgr::GetCurrentTopology [] 2021-08-16T13:47:08.205+00:00 [Info] ServiceMgr::GetCurrentTopology returns &{[0 0 0 0 0 0 0 0] [8a95cde683482bdec453edbb8b18eb5d] true []} 2021-08-16T13:47:08.206+00:00 [Info] ServiceMgr::GetTaskList [0 0 0 0 0 0 0 0] 2021-08-16T13:47:08.207+00:00 [Info] ServiceMgr::GetCurrentTopology [0 0 0 0 0 0 0 0] 2021-08-16T13:47:08.208+00:00 [Info] Indexer::updateStorageMode Try setting storage mode to plasma 2021-08-16T13:47:08.208+00:00 [Info] Indexer: Setting GC percent to 100 2021-08-16T13:47:08.208+00:00 [Info] CompactionManager: Refreshing settings 2021-08-16T13:47:08.208+00:00 [Info] MutationMgr::MaxQueueMemoryQuota 53687091 2021-08-16T13:47:08.208+00:00 [Info] ClustMgr:handleConfigUpdate 2021-08-16T13:47:08.210+00:00 [Info] lifecycleMgr.dispatchRequest: op OPCODE_CONFIG_UPDATE elapsed 1.269168ms len(expediates) 0 len(incomings) 0 len(outgoings) 0 len(parallels) 0 error 2021-08-16T13:47:08.212+00:00 [Info] updator: updating service map. server group=Group 1, indexerVersion=5 nodeAddr ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:8091 clusterVersion 5 excludeNode storageMode 2 2021-08-16T13:47:09.174+00:00 [Info] Indexer::ReadMemstats Time Taken 127.77µs 2021-08-16T13:47:11.670+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:47:11.706+00:00 [Info] ServiceMgr::PrepareTopologyChange {b82c752c1c8d25413a3ead033ffba421 [] topology-change-rebalance [{{8a95cde683482bdec453edbb8b18eb5d 5 } recovery-full}] []} 2021-08-16T13:47:11.709+00:00 [Info] Indexer::run:msg_loop: INDEXER_CHECK_DDL_IN_PROGRESS message from wrkrPrioRecvCh channel processing took 1.855µs 2021-08-16T13:47:11.709+00:00 [Info] ServiceMgr::prepareRebalance Found DDL Running [] 2021-08-16T13:47:11.709+00:00 [Info] ServiceMgr::prepareRebalance Init Prepare Phase 2021-08-16T13:47:11.709+00:00 [Info] ClustMgr:handleSetLocalValue Key RebalanceRunning Value 2021-08-16T13:47:11.710+00:00 [Info] ClustMgr:handleRebalanceRunning mType: CLUST_MGR_REBALANCE_RUNNING indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:47:11.710+00:00 [Info] Indexer::run:msg_loop: CLUST_MGR_SET_LOCAL message from wrkrPrioRecvCh channel processing took 1.322903ms 2021-08-16T13:47:11.710+00:00 [Info] ServiceMgr::PrepareTopologyChange Success. isBalanced true 2021-08-16T13:47:11.710+00:00 [Info] ServiceMgr::GetCurrentTopology returns &{[0 0 0 0 0 0 0 1] [8a95cde683482bdec453edbb8b18eb5d] true []} 2021-08-16T13:47:11.710+00:00 [Info] lifecycleMgr.dispatchRequest: op OPCODE_REBALANCE_RUNNING elapsed 2.511µs len(expediates) 0 len(incomings) 0 len(outgoings) 0 len(parallels) 0 error 2021-08-16T13:47:11.710+00:00 [Info] ServiceMgr::GetTaskList returns &{[0 0 0 0 0 0 0 1] [{[0 0 0 0 0 0 0 0] prepare/b82c752c1c8d25413a3ead033ffba421 task-prepared task-running true 0 map[] map[rebalanceId:b82c752c1c8d25413a3ead033ffba421]}]} 2021-08-16T13:47:11.711+00:00 [Info] ServiceMgr::GetCurrentTopology [0 0 0 0 0 0 0 1] 2021-08-16T13:47:11.711+00:00 [Info] ServiceMgr::GetTaskList [0 0 0 0 0 0 0 1] 2021-08-16T13:47:11.712+00:00 [Info] ServiceMgr::StartTopologyChange {b82c752c1c8d25413a3ead033ffba421 [] topology-change-rebalance [{{8a95cde683482bdec453edbb8b18eb5d 5 } recovery-full}] []} 2021-08-16T13:47:11.724+00:00 [Info] ClustMgr:handleSetLocalValue Key RebalanceToken Value {"MasterId":"8a95cde683482bdec453edbb8b18eb5d","RebalId":"b82c752c1c8d25413a3ead033ffba421","Source":0,"Error":"","MasterIP":"ec2-3-8-56-217.eu-west-2.compute.amazonaws.com"} 2021-08-16T13:47:11.725+00:00 [Info] ServiceMgr::registerLocalRebalanceToken Registered Rebalance Token In Local Meta &{8a95cde683482bdec453edbb8b18eb5d b82c752c1c8d25413a3ead033ffba421 Rebalance ec2-3-8-56-217.eu-west-2.compute.amazonaws.com} 2021-08-16T13:47:11.726+00:00 [Info] ServiceMgr::registerRebalanceTokenInMetakv Registered Global Rebalance Token In Metakv &{8a95cde683482bdec453edbb8b18eb5d b82c752c1c8d25413a3ead033ffba421 Rebalance ec2-3-8-56-217.eu-west-2.compute.amazonaws.com} 2021-08-16T13:47:11.733+00:00 [Info] ServiceMgr::registerGlobalRebalanceToken ClusterInfo Node List doesn't match Known Nodes in Rebalance Request. Skip Rebalance. cinfo.nodes : [{458753 active http://ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:8092/ ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:8091 map[] 6313 6313 7.691866112e+09 8.274788352e+09 aarch64-unknown-linux-gnu map[direct:11210 distTCP:21100 distTLS:21150 httpsCAPI:18092 httpsMgmt:18091] healthy 105 7.1.0-1152-enterprise true [backup fts index kv n1ql] 8a95cde683482bdec453edbb8b18eb5d inet}], change : {b82c752c1c8d25413a3ead033ffba421 [] topology-change-rebalance [{{8a95cde683482bdec453edbb8b18eb5d 5 } recovery-full}] []} 2021-08-16T13:47:11.733+00:00 [Info] ServiceMgr::startRebalance skip planner due to skipRebalance flag 2021-08-16T13:47:11.733+00:00 [Info] NewRebalancer nodeId 8a95cde683482bdec453edbb8b18eb5d rebalToken &{8a95cde683482bdec453edbb8b18eb5d b82c752c1c8d25413a3ead033ffba421 Rebalance ec2-3-8-56-217.eu-west-2.compute.amazonaws.com} master true localaddr 127.0.0.1:9102 runPlanner false runParam &{false []} 2021-08-16T13:47:11.733+00:00 [Info] ServiceMgr::StartTopologyChange returns Error . isBalanced true. 2021-08-16T13:47:11.733+00:00 [Info] ServiceMgr::GetCurrentTopology returns &{[0 0 0 0 0 0 0 2] [8a95cde683482bdec453edbb8b18eb5d] true []} 2021-08-16T13:47:11.733+00:00 [Info] ServiceMgr::GetTaskList returns &{[0 0 0 0 0 0 0 2] [{[0 0 0 0 0 0 0 0] prepare/b82c752c1c8d25413a3ead033ffba421 task-prepared task-running true 0 map[] map[rebalanceId:b82c752c1c8d25413a3ead033ffba421]} {[0 0 0 0 0 0 0 0] rebalance/b82c752c1c8d25413a3ead033ffba421 task-rebalance task-running true 0 map[] map[rebalanceId:b82c752c1c8d25413a3ead033ffba421]}]} 2021-08-16T13:47:11.734+00:00 [Info] ServiceMgr::GetTaskList [] 2021-08-16T13:47:11.734+00:00 [Info] ServiceMgr::GetTaskList returns &{[0 0 0 0 0 0 0 3] [{[0 0 0 0 0 0 0 0] prepare/b82c752c1c8d25413a3ead033ffba421 task-prepared task-running true 0 map[] map[rebalanceId:b82c752c1c8d25413a3ead033ffba421]} {[0 0 0 0 0 0 0 1] rebalance/b82c752c1c8d25413a3ead033ffba421 task-rebalance task-running true 1 map[] map[rebalanceId:b82c752c1c8d25413a3ead033ffba421]}]} 2021-08-16T13:47:11.735+00:00 [Info] ServiceMgr::GetCurrentTopology [] 2021-08-16T13:47:11.735+00:00 [Info] ServiceMgr::GetCurrentTopology returns &{[0 0 0 0 0 0 0 3] [8a95cde683482bdec453edbb8b18eb5d] true []} 2021-08-16T13:47:11.735+00:00 [Info] ServiceMgr::GetTaskList [0 0 0 0 0 0 0 3] 2021-08-16T13:47:11.735+00:00 [Info] ServiceMgr::GetCurrentTopology [0 0 0 0 0 0 0 3] 2021-08-16T13:47:11.736+00:00 [Info] Rebalancer::doFinish Cleanup 2021-08-16T13:47:11.736+00:00 [Info] ServiceMgr::runCleanupPhase path /indexing/rebalance/RebalanceToken isMaster true 2021-08-16T13:47:11.736+00:00 [Info] Rebalancer::processDropIndexQueue Done Received 2021-08-16T13:47:11.737+00:00 [Info] ServiceMgr::cleanupGlobalRToken Delete Global Rebalance Token {8a95cde683482bdec453edbb8b18eb5d b82c752c1c8d25413a3ead033ffba421 Rebalance ec2-3-8-56-217.eu-west-2.compute.amazonaws.com} 2021-08-16T13:47:11.740+00:00 [Info] ServiceMgr::cleanupLocalRToken Cleanup 2021-08-16T13:47:11.740+00:00 [Info] ClustMgr:handleDelLocalValue Key RebalanceToken 2021-08-16T13:47:11.741+00:00 [Info] ServiceMgr::cleanupRebalanceRunning Cleanup 2021-08-16T13:47:11.741+00:00 [Info] ClustMgr:handleDelLocalValue Key RebalanceRunning 2021-08-16T13:47:11.742+00:00 [Info] ServiceMgr::onRebalanceDoneLOCKED Rebalance Done: isBalanced true, isMaster true, forceUnbalanced false, err: 2021-08-16T13:47:11.742+00:00 [Info] ServiceMgr::GetTaskList returns &{[0 0 0 0 0 0 0 4] []} 2021-08-16T13:47:11.742+00:00 [Info] ServiceMgr::GetCurrentTopology returns &{[0 0 0 0 0 0 0 4] [8a95cde683482bdec453edbb8b18eb5d] true []} 2021-08-16T13:47:11.742+00:00 [Info] DDLServiceMgr: handling rebalance done 2021-08-16T13:47:11.743+00:00 [Info] ServiceMgr::GetTaskList [0 0 0 0 0 0 0 4] 2021-08-16T13:47:11.744+00:00 [Info] ServiceMgr::GetCurrentTopology [0 0 0 0 0 0 0 4] 2021-08-16T13:47:11.744+00:00 [Info] ServiceMgr::GetTaskList [] 2021-08-16T13:47:11.744+00:00 [Info] ServiceMgr::GetTaskList returns &{[0 0 0 0 0 0 0 4] []} 2021-08-16T13:47:11.744+00:00 [Info] ServiceMgr::GetCurrentTopology [] 2021-08-16T13:47:11.744+00:00 [Info] ServiceMgr::GetCurrentTopology returns &{[0 0 0 0 0 0 0 4] [8a95cde683482bdec453edbb8b18eb5d] true []} 2021-08-16T13:47:11.745+00:00 [Info] ServiceMgr::GetTaskList [0 0 0 0 0 0 0 4] 2021-08-16T13:47:11.745+00:00 [Info] ServiceMgr::GetCurrentTopology [0 0 0 0 0 0 0 4] 2021-08-16T13:47:11.748+00:00 [Error] DDLServiceMgr:rebalanceDone(): Failed to initialize metadata provider. Error=DDLServiceMgr:rebalanceDone: Failed to initialize metadata provider. Unknown host=map[8a95cde683482bdec453edbb8b18eb5d:true]. 2021-08-16T13:47:11.752+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:47:11.822+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:47:24.874+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:47:24.874+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:47:25.179+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:47:25.450+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:47:26.590+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:47:27.778+00:00 [Info] lifecycleMgr.dispatchRequest: op OPCODE_SERVICE_MAP elapsed 4.507159ms len(expediates) 0 len(incomings) 0 len(outgoings) 0 len(parallels) 0 error 2021-08-16T13:47:27.778+00:00 [Info] lifecycleMgr.dispatchRequest: op OPCODE_CLIENT_STATS elapsed 6.589µs len(expediates) 0 len(incomings) 0 len(outgoings) 0 len(parallels) 0 error 2021-08-16T13:47:27.783+00:00 [Info] [Queryport ":9101"] connection "172.31.29.9:49110" doReceive() ... 2021-08-16T13:47:27.857+00:00 [Info] LifecycleMgr.handlePrepareCreateIndex() : Accept 12750664089235961203 2021-08-16T13:47:27.857+00:00 [Info] lifecycleMgr.dispatchRequest: op OPCODE_PREPARE_CREATE_INDEX elapsed 157.151µs len(expediates) 0 len(incomings) 0 len(outgoings) 0 len(parallels) 0 error 2021-08-16T13:47:27.872+00:00 [Info] clustMgrAgent::OnIndexCreate Notification Received for Create Index DefnId: 12750664089235961203 Name: def_primary Using: plasma Bucket: travel-sample Scope/Id: _default/0 Collection/Id: _default/0 IsPrimary: true NumReplica: 0 InstVersion: 0 SecExprs: ([]) Desc: [] PartitionScheme: SINGLE HashScheme: CRC32 PartitionKeys: [] WhereExpr: () RetainDeletedXATTR: false &{0} partitions [0] 2021-08-16T13:47:27.872+00:00 [Info] Indexer::handleCreateIndex InstId: 633044510811627291 Defn: DefnId: 12750664089235961203 Name: def_primary Using: plasma Bucket: travel-sample Scope/Id: _default/0 Collection/Id: _default/0 IsPrimary: true NumReplica: 0 InstVersion: 0 SecExprs: ([]) Desc: [] PartitionScheme: SINGLE HashScheme: CRC32 PartitionKeys: [] WhereExpr: () RetainDeletedXATTR: false State: INDEX_STATE_CREATED RState: RebalActive Stream: NIL_STREAM Version: 0 ReplicaId: 0 PartitionContainer: &{map[0:{0 0 [:9105]}] 1024 1 1024 SINGLE 0} 2021-08-16T13:47:27.872+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:47:27.872+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:47:27.872+00:00 [Info] Directory /opt/couchbase/var/lib/couchbase/data/@2i/shards does not exist. Skip shard recovery. 2021-08-16T13:47:27.872+00:00 [Info] Shard /opt/couchbase/var/lib/couchbase/data/@2i/shards/shard1(1) : Shard Created Successfully unexpected fault address 0x4004c67b1f fatal error: fault 2021-08-16T13:47:27.872+00:00 [Info] Shard /opt/couchbase/var/lib/couchbase/data/@2i/shards/shard1(1) : metadata saved successfully [signal SIGBUS: bus error code=0x1 addr=0x4004c67b1f pc=0xbd8ea4] goroutine 479 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x40001069f0 sp=0x40001069c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4000106a30 sp=0x40001069f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x400543c820, 0x4005820000, 0x11a9890, 0x4000106b40, 0x400543c838, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4000106ab0 sp=0x4000106a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x400543c820, 0x4005820000, 0x11a9890, 0x4000106b40, 0x400543c838, 0x84, 0xc03eb8fbf4048cb0, 0x601d941fe) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4000106af0 sp=0x4000106ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4000033a40, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4000106da0 sp=0x4000106af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4000033a40, 0x400528f730, 0x68, 0x100, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4000106e60 sp=0x4000106da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x40056ed2b0, 0x40056ed2c0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4000107200 sp=0x4000106e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x40056ed2b0, 0x40056ed2c0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x40001078f0 sp=0x4000107200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x40058089f0, 0x40056e9500, 0x40051bb080, 0x40056ed290) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4000107fb0 sp=0x40001078f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4000107fb0 sp=0x4000107fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [semacquire]: sync.runtime_Semacquire(0x40058089f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x40058089f0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x40051bb080, 0x4004396d50, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xfffff8340969, 0x29, 0xfffff83409cb, 0x25, 0x4005728840, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4005621f70, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xfffff8340969, 0x29, 0xfffff83409cb, 0x25, 0x4005728840, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4005621f70, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x40001063d0, 0x40001060a0, 0x4004396d50, 0x40002f2900, 0x40001785c0, 0x1, 0x7f, 0x4000105fe8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x400133a000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4005621f70, 0xb, 0x112ff8f, 0x6, 0x4005621f80, 0xd, 0x40050e7100, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).handleCreateIndex(0x400133a000, 0x12da5c8, 0x4005770580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1690 +0x60c github.com/couchbase/indexing/secondary/indexer.(*indexer).handleAdminMsgs(0x400133a000, 0x12da5c8, 0x4005770580, 0x0, 0x4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1553 +0x154 github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x400133a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:992 +0x478 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x40000fb440, 0x4, 0x1136a87, 0xf, 0x4000107e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 7 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400000e690) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 8 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400000e6a8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 9 [IO wait]: internal/poll.runtime_pollWait(0xffff89a53098, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f8e18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f8e00, 0x400235b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f8e00, 0x400235b000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400133e008, 0x400235b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x4001334060, 0x4000348001, 0xdff, 0xdff, 0x40002958e8, 0x6902d8, 0x4000295948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400000e048, 0x4000348001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x4000342000, 0xa, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4000342000, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4000342000, 0xf5a960, 0x40013860e0, 0x6f0d60, 0x4004955e68) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x40013860c0, 0x4000046220, 0x4000295b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400008c050, 0x12f8f98, 0x40013860c0, 0x4004a5d617, 0x16, 0x40013cf7a0, 0x40000a3c01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400008c050, 0x12f8f98, 0x40013860c0, 0x400008c050, 0x400033a070, 0x400033a080, 0x40000f8000, 0x4000046020, 0xee0980, 0x40013cf7a0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400008c050, 0x12f8f98, 0x40013860c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x40000fa3c0, 0x40000a3f78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x4000295f78, 0x40000fa3c0, 0x12db8a8, 0x400000ed20, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x40000fa3c0, 0x40000b9180, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x40000fa3c0, 0x40000b9180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 10 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 11 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 12 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 13 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 14 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 15 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40000547a0, 0x4, 0x4000054668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 18 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400000ad00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 20 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4001354b80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 68 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400011e3c0, 0x12c18bc, 0x1, 0x40000a3e58, 0x40000385a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x400131aec8, 0x40000385a0, 0x4001314780, 0x4000038658) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x4714f4, 0x4000038600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40000a3fa0, 0x4001344000, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4000178900) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 66 [select]: net/http.(*persistConn).readLoop(0x400134e120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 67 [select]: net/http.(*persistConn).writeLoop(0x400134e120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 69 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4000178900) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 52 [IO wait]: internal/poll.runtime_pollWait(0xffff89a52fb0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4001356118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4001356100, 0x400235c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4001356100, 0x400235c000, 0x1000, 0x1000, 0x40013b8a88, 0x68f7a4, 0x40013ae050) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400133e010, 0x400235c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400134e120, 0x400235c000, 0x1000, 0x1000, 0xf12180, 0x40013b0040, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40013340c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40013340c0, 0xa, 0x1008880, 0x40013b0000, 0x199, 0x1000, 0x40013b8b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40013340c0, 0x2, 0x0, 0x9, 0xd5d, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400009ed50) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400009ed50, 0x4000336000, 0xe00, 0xe00, 0xb4, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40013a4000, 0x4000336000, 0xe00, 0xe00, 0xb4, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40013a4000, 0x4000336000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40013a4040, 0x4000336000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40013ae000, 0x0, 0x40013b8eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40013ae000, 0x0, 0x0, 0x40013b8f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40013ae000, 0xee0d40, 0x40013b0000, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4001394000, 0x4000082780, 0x40013ac000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 65 [select]: net/http.(*persistConn).readLoop(0x400428a000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 72 [IO wait]: internal/poll.runtime_pollWait(0xffff89a52ec8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400237e098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400237e080, 0x400001c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400237e080, 0x400001c000, 0x1000, 0x1000, 0x400236dbe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400133e060, 0x400001c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40002a4a20, 0x400001c000, 0x1000, 0x1000, 0x400236dcf8, 0x67867c, 0x4001336ba0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40013343c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40013343c0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40002a4a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 73 [select]: net/http.(*persistConn).writeLoop(0x40002a4a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 57 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001785c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 62 [IO wait]: internal/poll.runtime_pollWait(0xffff89a52870, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400237e318, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400237e300, 0x40002a1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400237e300, 0x40002a1000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40013cc4b8, 0x40002a1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40002a4fc0, 0x40002a1000, 0x1000, 0x1000, 0x400010d080, 0x200000003, 0x40000a77f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40013ecf00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40013ecf00, 0x40a, 0xffff604a0100, 0x100100000000, 0xffff6063afff, 0x400, 0x4004ca0000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40013ecf00, 0x41cff4, 0xffff604a0000, 0x100100000000, 0xffff6063afff, 0xffff60743b50) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000303230) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000303230, 0x400428d000, 0x1000, 0x1000, 0x40000a79b8, 0x0, 0x40000a7988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40013a5580, 0x400428d000, 0x1000, 0x1000, 0x4002347800, 0x3, 0x40000a7be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40013a5580, 0x400428d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40013a55c0, 0x400428d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40000a7dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000a7dd0, 0xa, 0x400428d201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40000a7dd0, 0x400428d20a, 0x1, 0x4004a5d7bb, 0x0, 0x400428d263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40000a7dd0, 0x4004c9210a, 0x4004a5d7bb, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40013c3a40, 0x400139fbc0, 0x24, 0x40000a7f20, 0x4001399b90, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x40013c3a40, 0x113075a, 0x7, 0x4001399b90, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x40013c3a00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 61 [IO wait]: internal/poll.runtime_pollWait(0xffff89a52788, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40013ead18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40013ead00, 0x4000329000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40013ead00, 0x4000329000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40013cc568, 0x4000329000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400428a000, 0x4000329000, 0x1000, 0x1000, 0x400010d200, 0x200000003, 0x400010d200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40013ed740) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40013ed740, 0x40013c400a, 0x400029b8c8, 0x4d3b10, 0x40000820c0, 0x40013c4000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40013ed740, 0x0, 0x1, 0x41a0a8, 0x40000820d8, 0x40013c4000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40023b4450) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40023b4450, 0x40023c1000, 0x1000, 0x1000, 0x400029b9b8, 0x0, 0x400029b988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000178a80, 0x40023c1000, 0x1000, 0x1000, 0x4002346600, 0x3, 0x400029bbe0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000178a80, 0x40023c1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000178ac0, 0x40023c1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400029bdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400029bdd0, 0xa, 0x40023c1101, 0x1, 0xe85, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400029bdd0, 0x40023c110a, 0x1, 0x40051f9969, 0x0, 0x40023c117b, 0x1, 0xe85, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400029bdd0, 0x40052ae20a, 0x40051f9969, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40013c3a40, 0x40013a3248, 0x17, 0x400029bf20, 0x4001399ba8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x40013c3a40, 0x113075a, 0x7, 0x4001399ba8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x40013c3a00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 108 [IO wait]: internal/poll.runtime_pollWait(0xffff6075ca98, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f8698, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f8680, 0x40042da000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f8680, 0x40042da000, 0x1000, 0x1000, 0x40042cfbe0, 0x469b10, 0x4004917800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000105a0, 0x40042da000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40013ce240, 0x40042da000, 0x1000, 0x1000, 0x40042cfcf8, 0x67867c, 0x40042d81e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40042a4720) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40042a4720, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40013ce240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 59 [select]: net/http.(*persistConn).readLoop(0x40002a4fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 60 [select]: net/http.(*persistConn).writeLoop(0x40002a4fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 76 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x40023801c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 77 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x40023801c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 78 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x40023801c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 79 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x40023801c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 88 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000179000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 89 [IO wait]: internal/poll.runtime_pollWait(0xffff89a526a0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40013eaf18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40013eaf00, 0x4004290000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40013eaf00, 0x4004290000, 0x1000, 0x1000, 0xe, 0x400033ec00, 0x23) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40013cc588, 0x4004290000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400428a120, 0x4004290000, 0x1000, 0x1000, 0x40003563b0, 0x40003562c0, 0x4001386158) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40013edaa0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40013edaa0, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40013edaa0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40013388d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40013388d0, 0x40023f5600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4001355640, 0x40023f5600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4001355640, 0x40023f5600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4001355680, 0x40023f5600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40023fd600, 0x40013dfe00, 0x4004364eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40023fd600, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40023fd600, 0xee0d40, 0x400135eaf0, 0x12db068, 0x4001355680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4001394f30, 0x4001386a80, 0x40013ac420) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 119 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x40013e6640, 0x40013dfd40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 98 [select]: net/http.(*persistConn).writeLoop(0x400428a000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 101 [select]: net/http.(*persistConn).readLoop(0x400428a120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 102 [select]: net/http.(*persistConn).writeLoop(0x400428a120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 37 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000178840) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 38 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x40002b7da0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 39 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x40002f89f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 40 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x4001356200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 41 [IO wait]: internal/poll.runtime_pollWait(0xffff89a525b8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4001356318, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4001356300, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4001356300, 0x4004dbbe60, 0x400236ae88, 0x445c24) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40013523f0, 0x400236aef0, 0x400236aef8, 0x18) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40013523f0, 0x11a95b0, 0x4001356280, 0x1303898, 0x400135af68) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x4001356280) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 42 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40000a04e0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 43 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40000a04e0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 44 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x40000a04e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 45 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x40002f94d0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 183 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x40048b0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:752 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 145 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40000f0380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 48 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x4000176770) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 178 [IO wait]: internal/poll.runtime_pollWait(0xffff6075c7e0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40013eb098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40013eb080, 0x4000337000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40013eb080, 0x4000337000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40013cc5a0, 0x4000337000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400428b0e0, 0x4000337000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40023952c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40023952c0, 0x100a, 0x0, 0x0, 0x400235ebe8, 0x597c30, 0x400235ec38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40023952c0, 0x412b8c, 0x400235ec78, 0x655784, 0x400235eca8, 0x40b2f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004902480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004902480, 0x40023f5a00, 0x200, 0x200, 0x4, 0x0, 0x400235ed01) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4001355bc0, 0x40023f5a00, 0x200, 0x200, 0x40013acae7, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4001355bc0, 0x40023f5a00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4001355c00, 0x40023f5a00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004900840, 0x680d00, 0x400235eeb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004900840, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004900840, 0xee0d40, 0x400135ed20, 0x12db068, 0x4001355c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40013955f0, 0x4001387440, 0x40013acea0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 21 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40013a4380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 22 [IO wait]: internal/poll.runtime_pollWait(0xffff89a52958, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40013ea318, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40013ea300, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40013ea300, 0x98f764, 0x1, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x400033cb70, 0x400133ec10, 0x1303898, 0x400133ec10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x400033cb70, 0x1303898, 0x400133ec10, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x40013a4d00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 23 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x40013a4ec0, 0x40013a8930, 0x40013a89a0, 0x4004288f00, 0x12e8f90, 0x40000f0000, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x40013960d4, 0x5, 0x40013a4d00, 0x12eca70, 0x40013a4b80, 0x130d478, 0x40000e80e0, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40000f01c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40000f01c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 24 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*metaNotifier).OnIndexCreate(0x400033cb88, 0x4004e492c0, 0x8c906a0e9f01f1b, 0x0, 0x4005672040, 0x1, 0x1, 0x4005672048, 0x1, 0x1, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:708 +0x260 github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).CreateIndex(0x40000f0000, 0x4004e492c0, 0x4005672000, 0x4005672000, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:1463 +0xaac github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).CreateIndexOrInstance(0x40000f0000, 0x4004e492c0, 0x400565e200, 0x4005672000, 0x4005647c00, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:1370 +0x1d0 github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processCommitToken(0x40000f0000, 0xb0f3756ee93d3573, 0x4005644de0, 0x0, 0x0, 0x12da228, 0x4005644e10, 0x1f0, 0x40056525a8, 0x400564d600, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:932 +0x1b8 github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).handleCommitCreateIndex(0x40000f0000, 0x40056546c0, 0x4004d57c78, 0x40056546c0, 0x0, 0x0, 0x1ba2100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:737 +0x1ac github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).handleCommit(0x40000f0000, 0x400564d600, 0x1f0, 0x200, 0x4004d57e28, 0x3, 0x3, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:690 +0x32c github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).dispatchRequest(0x40000f0000, 0x400566a280, 0x4004d57f10) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:497 +0xd34 github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40000f0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:371 +0x324 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 25 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40000f0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2659 +0x160 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 26 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x4002399710, 0x400035b6e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 27 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x4000178fc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 28 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x40013e6640) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 29 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x40013e6640) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 30 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x40013cc1d8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 31 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x40013e66e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 32 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x40013e66e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 33 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x40013a4ec0, 0x4004288f00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 120 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400011e3c0, 0x1144559, 0x1b, 0x4000299e38, 0x40013dfe60, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x400006ce90, 0x40013dfe60, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000299f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40023db080, 0x400006ce90) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 121 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 122 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400237aa00, 0xd00007, 0x40023d77e8, 0x5, 0x400006ceb0, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 123 [IO wait]: internal/poll.runtime_pollWait(0xffff89a523e8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4001356818, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4001356800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4001356800, 0x0, 0x40013b2d88, 0x665fe4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4001398558, 0x611a6c62, 0x40013b2db8, 0x445c24) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4001398558, 0x40013b2e20, 0x40013b2e28, 0x18, 0x4004394480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40023800e0, 0x12ed9d0, 0x4001398558, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40023800e0, 0x12ed9d0, 0x4001398558, 0x400133a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 124 [IO wait]: internal/poll.runtime_pollWait(0xffff89a52300, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4001356918, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4001356900, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4001356900, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4001398570, 0xffff89ba1a68, 0xffff6071b578, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4001398570, 0x665e1c, 0x0, 0xfe5e60, 0x40013b3dc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4001398588, 0x4004396030, 0x11ae500, 0x40023805c0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x4002380540, 0x12ec710, 0x4001398588, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x4002380540, 0x12ec710, 0x4001398588, 0x400133a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 182 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x40048b0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:662 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 104 [IO wait]: internal/poll.runtime_pollWait(0xffff6075cb80, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f8598, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f8580, 0x40042d0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f8580, 0x40042d0000, 0x1000, 0x1000, 0x1900e48a127, 0x0, 0x40000a5a98) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010550, 0x40042d0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*connReader).Read(0x400429c7b0, 0x40042d0000, 0x1000, 0x1000, 0xab8f79a20, 0x1be4a80, 0x72) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:800 +0x23c bufio.(*Reader).fill(0x40042a43c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40042a43c0, 0x4, 0xab8f79a20, 0x1be4a80, 0x0, 0x0, 0x1be4a80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*conn).serve(0x40000b9220, 0x12f86a0, 0x4000081b40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:1977 +0x840 created by net/http.(*Server).Serve /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:3013 +0x318 goroutine 181 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x40048b0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:612 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 90 [IO wait]: internal/poll.runtime_pollWait(0xffff89a52218, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400237eb98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400237eb80, 0x40023f9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400237eb80, 0x40023f9000, 0x1000, 0x1000, 0x191799c82e0, 0x0, 0x40048a1a98) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400135a280, 0x40023f9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*connReader).Read(0x4001338b70, 0x40023f9000, 0x1000, 0x1000, 0xc2448a9ed, 0x1be4a80, 0x72) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:800 +0x23c bufio.(*Reader).fill(0x4001386c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4001386c00, 0x4, 0xc2448a9ed, 0x1be4a80, 0x0, 0x0, 0x1be4a80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*conn).serve(0x4001361220, 0x12f86a0, 0x40013556c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:1977 +0x840 created by net/http.(*Server).Serve /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:3013 +0x318 goroutine 179 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40048b2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 109 [select]: net/http.(*persistConn).writeLoop(0x40013ce240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 92 [IO wait]: internal/poll.runtime_pollWait(0xffff6075c8c8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40013ea798, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40013ea780, 0x40023ff000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40013ea780, 0x40023ff000, 0x1000, 0x1000, 0x40023bebe0, 0x469b10, 0x4002347380) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400135a368, 0x40023ff000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400428ad80, 0x40023ff000, 0x1000, 0x1000, 0x40023becf8, 0x67867c, 0x40013ac660) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4001386d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4001386d80, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400428ad80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 228 [select]: net/http.(*persistConn).writeLoop(0x400428b200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 93 [select]: net/http.(*persistConn).writeLoop(0x400428ad80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 146 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40000f0380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 147 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40000f0380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 148 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40000f0380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 149 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x40013a5fc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 166 [select]: net/http.(*persistConn).writeLoop(0x400000db00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 152 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40048b0160, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 153 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40048b0160, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 154 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x40048b0160) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 180 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x40048b0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:224 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 157 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40000f0380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 158 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400011e3c0, 0x1144559, 0x1b, 0x40048c3e38, 0x4002382c00, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004289a50, 0x4002382c00, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40048c3f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40023943c0, 0x4004289a50) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 161 [select]: net/http.(*persistConn).readLoop(0x400428b0e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 162 [select]: net/http.(*persistConn).writeLoop(0x400428b0e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 184 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4000359300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:249 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 186 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x400133a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9716 +0x474 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 187 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x400133a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1090 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 188 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x400133a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1025 +0xfc created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 339 [select]: net/http.(*persistConn).writeLoop(0x4004d3a360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 191 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).processSchedIndexes(0x40048b0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:296 +0x44 created by github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:218 +0x3c goroutine 192 [IO wait]: internal/poll.runtime_pollWait(0xffff6075c610, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400237f218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400237f200, 0x40043ab000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400237f200, 0x40043ab000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400133e300, 0x40043ab000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x40023db680, 0x40048e2601, 0x5ff, 0x5ff, 0x400239f888, 0x6902d8, 0x400239f8e8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x4001398a20, 0x40048e2601, 0x5ff, 0x5ff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x400439f1e0, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400439f1e0, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400439f1e0, 0xf5a960, 0x40023dbc40, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x40023dbc20, 0x4002375fa0, 0x400239fac8, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400237ac30, 0x12f8f98, 0x40023dbc20, 0x4004b720a0, 0x16, 0x199, 0x400492fc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400237ac30, 0x12f8f98, 0x40023dbc20, 0x400237ac30, 0x40023d7ef8, 0x40023d7f00, 0x4001356b00, 0x40013ab540, 0xfb91e0, 0x4002375f80, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400237ac30, 0x12f8f98, 0x40023dbc20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4004903470, 0x400492ff10, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400239ff10, 0x4004903470, 0x0, 0x0, 0x0, 0xd8d03c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth/service.RegisterManager(0x13007d8, 0x4000359300, 0x0, 0x0, 0x40023b9f80, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:146 +0xa0 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).registerWithServer(0x4000359300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:301 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:205 +0x78 goroutine 328 [IO wait]: internal/poll.runtime_pollWait(0xffff6075bfb8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004ce0118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004ce0100, 0x4004cf4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004ce0100, 0x4004cf4000, 0x1000, 0x1000, 0x19179a15b4f, 0x0, 0x4004cffa98) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400135a980, 0x4004cf4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*connReader).Read(0x4004cdae70, 0x4004cf4000, 0x1000, 0x1000, 0xc244798c1, 0x1be4a80, 0x72) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:800 +0x23c bufio.(*Reader).fill(0x4004cd2660) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004cd2660, 0x4, 0xc244798c1, 0x1be4a80, 0x0, 0x0, 0x1be4a80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*conn).serve(0x4004a595e0, 0x12f86a0, 0x4004cde580) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:1977 +0x840 created by net/http.(*Server).Serve /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:3013 +0x318 goroutine 196 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*janitor).run(0x4001385b30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:4377 +0xac created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).OnNewRequest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:244 +0x2e8 goroutine 197 [select]: github.com/couchbase/indexing/secondary/manager.(*builder).run.func1(0x4001385980) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:4452 +0xa0 github.com/couchbase/indexing/secondary/manager.(*builder).run(0x4001385980) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:4459 +0xa4 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).OnNewRequest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:247 +0x30c goroutine 198 [select]: github.com/couchbase/indexing/secondary/manager.(*updator).run(0x40013b0280) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:4902 +0x148 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).OnNewRequest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:250 +0x330 goroutine 199 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processParallelsRequests(0x40000f0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:408 +0x10c created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).OnNewRequest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:253 +0x350 goroutine 203 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400011e3c0, 0x1144559, 0x1b, 0x4004931e38, 0x400035b560, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4000357560, 0x400035b560, 0x11ae420, 0x400004a800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0xd96838, 0x40042c6f38) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004931f80, 0xd96838, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40013ecc60, 0x4000357560) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 201 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400011e3c0, 0x113ba6f, 0x14, 0x400493ff38, 0x4002383d40, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x113ba6f, 0x14, 0x400493ff98, 0x4002383d40, 0x40048c8fb8, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).listenMoveIndex(0x4000359300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:2297 +0xd8 created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).recoverRebalance /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:1966 +0x88 goroutine 202 [sleep]: time.Sleep(0x6fc23ac00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).rebalanceJanitor(0x4000359300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:1287 +0x40 created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).recoverRebalance /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:1967 +0xa8 goroutine 230 [IO wait]: internal/poll.runtime_pollWait(0xffff6075c6f8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40013eb418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40013eb400, 0x40048eb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40013eb400, 0x40048eb000, 0x1000, 0x1000, 0x1909ec02ceb, 0x0, 0x4004297a98) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40013cc7e8, 0x40048eb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*connReader).Read(0x4004a4e1e0, 0x40048eb000, 0x1000, 0x1000, 0xb4971dbc3, 0x1be4a80, 0x72) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:800 +0x23c bufio.(*Reader).fill(0x40048f0720) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40048f0720, 0x4, 0xb4971dbc3, 0x1be4a80, 0x0, 0x0, 0x1be4a80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*conn).serve(0x40013e7900, 0x12f86a0, 0x40049ef2c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:1977 +0x840 created by net/http.(*Server).Serve /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:3013 +0x318 goroutine 167 [select]: net/http.(*persistConn).readLoop(0x400000d9e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 206 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400011e3c0, 0x1144559, 0x1b, 0x4004941e38, 0x400035b4a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4000357610, 0x400035b4a0, 0x4001319728, 0xd8d094) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x4000000004) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004941f80, 0x4714f4, 0x1be18c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40013ecc00, 0x4000357610) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 168 [select]: net/http.(*persistConn).writeLoop(0x400000d9e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 165 [select]: net/http.(*persistConn).readLoop(0x400000db00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 293 [select]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).wait(0x4000359300, 0x4004abad00, 0x8, 0x9, 0x4004ad0480, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:1807 +0xf4 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).GetTaskList(0x4000359300, 0x4004abad00, 0x8, 0x9, 0x4004ad0480, 0xadb658, 0x40048dda28, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:338 +0xd4 github.com/couchbase/cbauth/service.serviceAPI.GetTaskList.func1(0x4004ad0480) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:99 +0x68 github.com/couchbase/cbauth/service.withTimeout(0x7530, 0x40048ddb00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:46 +0x6c github.com/couchbase/cbauth/service.serviceAPI.GetTaskList(0x13007d8, 0x4000359300, 0x4004abad00, 0x8, 0x9, 0x7530, 0x4004ac6a80, 0x46b478, 0x4004abc708) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:98 +0x94 reflect.Value.call(0x40023dbaa0, 0x400133e398, 0x13, 0x112ec54, 0x4, 0x40048ddf00, 0x3, 0x3, 0x400235feb8, 0x6a1f00, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/reflect/value.go:476 +0x644 reflect.Value.Call(0x40023dbaa0, 0x400133e398, 0x13, 0x400235ff00, 0x3, 0x3, 0x4004aa0b00, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/reflect/value.go:337 +0x84 net/rpc.(*service).call(0x4000179c40, 0x400237ac30, 0x40023d7ef8, 0x40023d7f00, 0x4001356c00, 0x40013ab740, 0xfb9280, 0x4004ab4500, 0x199, 0xee10c0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:377 +0x13c created by net/rpc.(*Server).ServeCodec /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:474 +0x2f8 goroutine 348 [IO wait]: internal/poll.runtime_pollWait(0xffff6075c0a0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004ce0298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004ce0280, 0x4004e05000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004ce0280, 0x4004e05000, 0x1000, 0x1000, 0x4004935688, 0x69f178, 0x40049356f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40013cccf0, 0x4004e05000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004caf7a0, 0x4004e05000, 0x1000, 0x1000, 0x5, 0x4005686f00, 0x40049357a8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004d72fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004d72fc0, 0x40a, 0xffff604f5e00, 0x100100000000, 0xffff6063afff, 0x400, 0x4004935810) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004d72fc0, 0x41cff4, 0xffff604f5d00, 0x100100000000, 0xffff6063afff, 0xffff603d3f30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004d9a5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004d9a5a0, 0x4004da0000, 0x1000, 0x1000, 0x4004935968, 0x0, 0x4004935938) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004cdec80, 0x4004da0000, 0x1000, 0x1000, 0x4002346600, 0x3, 0x4004935b90) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004cdec80, 0x4004da0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004cdecc0, 0x4004da0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4004935d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004935d80, 0xa, 0x4004da0201, 0x1, 0xda7, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4004935d80, 0x4004da020a, 0x1, 0x40056f534e, 0x0, 0x4004da0259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4004935d80, 0x400571ec0a, 0x40056f534e, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40013c3a40, 0x4004961ca0, 0x1f, 0x4004935ee0, 0x4004c9bec0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x40013c3a40, 0x113075a, 0x7, 0x4004d74270, 0xd, 0x4004c9bec0, 0x0, 0xfb2801, 0x1c1bd01) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x40013c3a00, 0x4004d74270, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 227 [select]: net/http.(*persistConn).readLoop(0x400428b200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 211 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorMemUsage(0x400133a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8873 +0x11c created by github.com/couchbase/indexing/secondary/indexer.(*indexer).handleStorageWarmupDone /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6788 +0x534 goroutine 212 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*indexer).logMemstats(0x400133a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9159 +0x12c created by github.com/couchbase/indexing/secondary/indexer.(*indexer).handleStorageWarmupDone /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6789 +0x554 goroutine 216 [IO wait]: internal/poll.runtime_pollWait(0xffff6075c528, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400237f318, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400237f300, 0x40048d5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400237f300, 0x40048d5000, 0x1000, 0x1000, 0x40042cca38, 0x46b37c, 0x40042ccac8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40013cc5b8, 0x40048d5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000d9e0, 0x40048d5000, 0x1000, 0x1000, 0x0, 0x400490e2a0, 0x40042ccbe8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002395620) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002395620, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4002395620, 0x41ab8c, 0xffff60485b00, 0x100100000000, 0xffff6063afff, 0x400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004903c20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004903c20, 0x4004910600, 0x200, 0x200, 0x40048ba508, 0x41cff4, 0xffff60485b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40049460c0, 0x4004910600, 0x200, 0x200, 0xffff6071ccd8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40049460c0, 0x4004910600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004946100, 0x4004910600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004956000, 0x0, 0x40048ba6b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004956000, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004956000, 0xee0d40, 0x400135f540, 0x12db068, 0x4004946100) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004936900, 0x4004942f60, 0x4004938a20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 214 [sleep]: time.Sleep(0xd18c2e2800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsPersister(0x40013e6640) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3241 +0xe0 created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3035 +0x248 goroutine 217 [IO wait]: internal/poll.runtime_pollWait(0xffff6075c270, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40013eb198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40013eb180, 0x40043b8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40013eb180, 0x40043b8000, 0x1000, 0x1000, 0x40042cda88, 0x68f7a4, 0x40049561b0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400133e318, 0x40043b8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400428b200, 0x40043b8000, 0x1000, 0x1000, 0xf12180, 0x400135f5d0, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40023db8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40023db8c0, 0xa, 0x1008880, 0x400135f590, 0x199, 0x1000, 0x40042cdb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40023db8c0, 0x2, 0x0, 0x9, 0x1ba, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004903c80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004903c80, 0x4004910800, 0x200, 0x200, 0x57, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004946140, 0x4004910800, 0x200, 0x200, 0x57, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004946140, 0x4004910800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004946180, 0x4004910800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004956160, 0x0, 0x40042cdeb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004956160, 0x0, 0x0, 0x40042cdf70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004956160, 0xee0d40, 0x400135f590, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004936990, 0x4004942fc0, 0x4004938a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 285 [select]: github.com/couchbase/cbauth/service.withTimeout.func1(0x7530, 0x4004b4d0d0, 0x4004b5ec60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:38 +0x78 created by github.com/couchbase/cbauth/service.withTimeout /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:36 +0xd8 goroutine 173 [IO wait]: internal/poll.runtime_pollWait(0xffff6075c440, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400237f418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400237f400, 0x40048d3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400237f400, 0x40048d3000, 0x1000, 0x1000, 0x40048e4360, 0xadb658, 0x40048c8a28) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40013cc5b0, 0x40048d3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000db00, 0x40048d3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002395500) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002395500, 0xa, 0x0, 0x0, 0x0, 0x0, 0x40042c43c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4002395500, 0xee10c0, 0x40042c4401, 0x655784, 0x40042c4608, 0x498444) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40048ab5f0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40048ab5f0, 0x4000317800, 0x200, 0x200, 0x4, 0x0, 0x2) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40048b23c0, 0x4000317800, 0x200, 0x200, 0x40042c45e8, 0x412bc0, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40048b23c0, 0x4000317800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40048b2400, 0x4000317800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40048d8840, 0x45a300, 0x40042c46b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40048d8840, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40048d8840, 0xee0d40, 0x40013b16d0, 0x12db068, 0x40048b2400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400238f320, 0x4002395da0, 0x40048e4420) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 284 [select]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).wait(0x4000359300, 0x4004b4d0c0, 0x8, 0x9, 0x4004b5ec60, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:1807 +0xf4 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).GetCurrentTopology(0x4000359300, 0x4004b4d0c0, 0x8, 0x9, 0x4004b5ec60, 0xadb658, 0x40043b6a28, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:390 +0xd4 github.com/couchbase/cbauth/service.serviceAPI.GetCurrentTopology.func1(0x4004b5ec60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:76 +0x68 github.com/couchbase/cbauth/service.withTimeout(0x7530, 0x40043b6b00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:46 +0x6c github.com/couchbase/cbauth/service.serviceAPI.GetCurrentTopology(0x13007d8, 0x4000359300, 0x4004b4d0c0, 0x8, 0x9, 0x7530, 0x400237bea0, 0x46b478, 0x40042e3688) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:75 +0x94 reflect.Value.call(0x40023db9e0, 0x400133e358, 0x13, 0x112ec54, 0x4, 0x40043b6f00, 0x3, 0x3, 0x2, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/reflect/value.go:476 +0x644 reflect.Value.Call(0x40023db9e0, 0x400133e358, 0x13, 0x40042c4700, 0x3, 0x3, 0x40042a5500, 0x40048add08, 0x40042c4784) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/reflect/value.go:337 +0x84 net/rpc.(*service).call(0x4000179c40, 0x400237ac30, 0x40023d7ef8, 0x40023d7f00, 0x4001356b00, 0x4002374e80, 0xfb91e0, 0x4002375f40, 0x199, 0xee1180, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:377 +0x13c created by net/rpc.(*Server).ServeCodec /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:474 +0x2f8 goroutine 338 [IO wait]: internal/poll.runtime_pollWait(0xffff89a524d0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40013ebc98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40013ebc80, 0x4004d62000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40013ebc80, 0x4004d62000, 0x1000, 0x1000, 0x4004b19be0, 0x469b10, 0x4002347680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40013ccb98, 0x4004d62000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004d3a360, 0x4004d62000, 0x1000, 0x1000, 0x4004b19cf8, 0x67867c, 0x4004d60120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004ad39e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004ad39e0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004d3a360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 288 [select]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).wait(0x4000359300, 0x4004b4d160, 0x8, 0x9, 0x4004b5ed80, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:1807 +0xf4 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).GetCurrentTopology(0x4000359300, 0x4004b4d160, 0x8, 0x9, 0x4004b5ed80, 0xadb658, 0x40043b6a28, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:390 +0xd4 github.com/couchbase/cbauth/service.serviceAPI.GetCurrentTopology.func1(0x4004b5ed80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:76 +0x68 github.com/couchbase/cbauth/service.withTimeout(0x7530, 0x40043b6b00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:46 +0x6c github.com/couchbase/cbauth/service.serviceAPI.GetCurrentTopology(0x13007d8, 0x4000359300, 0x4004b4d160, 0x8, 0x9, 0x7530, 0x4004b720a0, 0x46b478, 0x40042e3748) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:75 +0x94 reflect.Value.call(0x40023db9e0, 0x400133e358, 0x13, 0x112ec54, 0x4, 0x40043b6f00, 0x3, 0x3, 0x2, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/reflect/value.go:476 +0x644 reflect.Value.Call(0x40023db9e0, 0x400133e358, 0x13, 0x4004366700, 0x3, 0x3, 0x40043666e8, 0x4bcf10, 0x4004b72058) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/reflect/value.go:337 +0x84 net/rpc.(*service).call(0x4000179c40, 0x400237ac30, 0x40023d7ef8, 0x40023d7f00, 0x4001356b00, 0x40013ab540, 0xfb91e0, 0x4002375f80, 0x199, 0xee1180, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:377 +0x13c created by net/rpc.(*Server).ServeCodec /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:474 +0x2f8 goroutine 289 [select]: github.com/couchbase/cbauth/service.withTimeout.func1(0x7530, 0x4004b4d170, 0x4004b5ed80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:38 +0x78 created by github.com/couchbase/cbauth/service.withTimeout /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:36 +0xd8 goroutine 376 [select]: net/http.(*persistConn).writeLoop(0x4004caf7a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 467 [select]: net/http.(*persistConn).writeLoop(0x40052fcc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 298 [select]: github.com/couchbase/cbauth/service.withTimeout.func1(0x7530, 0x4004abada0, 0x4004ad05a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:38 +0x78 created by github.com/couchbase/cbauth/service.withTimeout /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:36 +0xd8 goroutine 294 [select]: github.com/couchbase/cbauth/service.withTimeout.func1(0x7530, 0x4004abad10, 0x4004ad0480) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:38 +0x78 created by github.com/couchbase/cbauth/service.withTimeout /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:36 +0xd8 goroutine 297 [select]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).wait(0x4000359300, 0x4004abad90, 0x8, 0x9, 0x4004ad05a0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:1807 +0xf4 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).GetTaskList(0x4000359300, 0x4004abad90, 0x8, 0x9, 0x4004ad05a0, 0xadb658, 0x40048dda28, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:338 +0xd4 github.com/couchbase/cbauth/service.serviceAPI.GetTaskList.func1(0x4004ad05a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:99 +0x68 github.com/couchbase/cbauth/service.withTimeout(0x7530, 0x40048ddb00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:46 +0x6c github.com/couchbase/cbauth/service.serviceAPI.GetTaskList(0x13007d8, 0x4000359300, 0x4004abad90, 0x8, 0x9, 0x7530, 0x4004ac6b70, 0x46b478, 0x4004abc7c8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:98 +0x94 reflect.Value.call(0x40023dbaa0, 0x400133e398, 0x13, 0x112ec54, 0x4, 0x40048ddf00, 0x3, 0x3, 0x2, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/reflect/value.go:476 +0x644 reflect.Value.Call(0x40023dbaa0, 0x400133e398, 0x13, 0x400235ff00, 0x3, 0x3, 0x400235fee8, 0x4bcf10, 0x4004aac6e8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/reflect/value.go:337 +0x84 net/rpc.(*service).call(0x4000179c40, 0x400237ac30, 0x40023d7ef8, 0x40023d7f00, 0x4001356c00, 0x40023752c0, 0xfb9280, 0x4004ab4540, 0x199, 0xee10c0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:377 +0x13c created by net/rpc.(*Server).ServeCodec /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:474 +0x2f8 goroutine 375 [select]: net/http.(*persistConn).readLoop(0x4004caf7a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 414 [select]: github.com/couchbase/indexing/secondary/queryport.(*Server).doPing(0x4001356280, 0x40051226c0, 0x4005113440) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:348 +0xf8 created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:265 +0x1c0 goroutine 453 [select]: net/http.(*persistConn).writeLoop(0x4004c2d7a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 466 [IO wait]: internal/poll.runtime_pollWait(0xffff60688a98, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004c2f898, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004c2f880, 0x4005723000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004c2f880, 0x4005723000, 0x1000, 0x1000, 0x40048c9be0, 0x469b10, 0x4002346600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400133ec00, 0x4005723000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40052fcc60, 0x4005723000, 0x1000, 0x1000, 0x40048c9cf8, 0x67867c, 0x400555dec0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40056bf2c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40056bf2c0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40052fcc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 452 [IO wait]: internal/poll.runtime_pollWait(0xffff6075c188, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004c2ed98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004c2ed80, 0x40052cd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004c2ed80, 0x40052cd000, 0x1000, 0x1000, 0x4004b18be0, 0x469b10, 0x40048f3800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400133e9c8, 0x40052cd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004c2d7a0, 0x40052cd000, 0x1000, 0x1000, 0x4004b18cf8, 0x67867c, 0x4005113260) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40052c2a80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40052c2a80, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004c2d7a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 470 [chan receive]: github.com/couchbase/gometa/common.(*PeerPipe).doSend(0x400547d080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:165 +0x64 created by github.com/couchbase/gometa/common.NewPeerPipe /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:63 +0x114 goroutine 413 [IO wait]: internal/poll.runtime_pollWait(0xffff6075c9b0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400515a518, 0x72, 0x0, 0x10000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400515a500, 0x4005160000, 0x6, 0x10000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400515a500, 0x4005160000, 0x6, 0x10000, 0x4005055180, 0x400515a580, 0x4005160000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400135af68, 0x4005160000, 0x6, 0x10000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 github.com/couchbase/indexing/secondary/transport.fullRead(0xffff60688d58, 0x400135af68, 0x4005160000, 0x6, 0x10000, 0x4005160000, 0x6) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:189 +0x90 github.com/couchbase/indexing/secondary/transport.Receive(0xffff60688d58, 0x400135af68, 0x4005160000, 0x10000, 0x10000, 0x0, 0x60, 0x0, 0x400492fdf8, 0x4108e0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/util.go:60 +0x60 github.com/couchbase/indexing/secondary/transport.(*TransportPacket).Receive(0x400492ff78, 0xffff60688d58, 0x400135af68, 0xffff60688d58, 0x400135af68, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/transport/transport.go:119 +0x4c github.com/couchbase/indexing/secondary/queryport.(*Server).doReceive(0x4001356280, 0x1303898, 0x400135af68, 0x40051226c0, 0x4005113440) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:300 +0x25c created by github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:264 +0x190 goroutine 412 [chan receive]: github.com/couchbase/indexing/secondary/queryport.(*Server).handleConnection(0x4001356280, 0x1303898, 0x400135af68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:272 +0x1f8 created by github.com/couchbase/indexing/secondary/queryport.(*Server).listener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:224 +0x2d4 goroutine 471 [IO wait]: internal/poll.runtime_pollWait(0xffff6075bd00, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004c2fd98, 0x72, 0x0, 0x8, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004c2fd80, 0x4005621d90, 0x8, 0x8, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004c2fd80, 0x4005621d90, 0x8, 0x8, 0x0, 0x736aa8, 0x4004e0fe08) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400133ec10, 0x4005621d90, 0x8, 0x8, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 github.com/couchbase/gometa/common.(*PeerPipe).readBytes(0x400547d080, 0x8, 0x0, 0x0, 0x0, 0x1, 0x0, 0x298, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:284 +0x80 github.com/couchbase/gometa/common.(*PeerPipe).doReceive(0x400547d080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:213 +0xb0 created by github.com/couchbase/gometa/common.NewPeerPipe /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerPipe.go:64 +0x134 goroutine 478 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 goroutine 475 [select]: github.com/couchbase/gometa/protocol.(*messageListener).start(0x4005565770) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:434 +0x164 created by github.com/couchbase/gometa/protocol.(*Leader).AddWatcher /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:276 +0x4c8 goroutine 477 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*metaNotifier).fetchStats(0x400033cb88) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:898 +0xa0 created by github.com/couchbase/indexing/secondary/indexer.(*metaNotifier).OnFetchStats /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:887 +0x3c goroutine 480 [runnable]: github.com/couchbase/plasma.(*StatsLogger).run(0x40000c5080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/stats.go:1281 created by github.com/couchbase/plasma.(*StatsLogger).startOrStop /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/stats.go:1182 +0xfc Initializing write barrier = 8000 2021-08-16T13:47:28.066+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:47:28.066+00:00 [Info] Setting ipv6=false 2021-08-16T13:47:28.074+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:47:28.079+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:47:28.079+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:47:28.079+00:00 [Info] Setting log level to Info 2021-08-16T13:47:28.079+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:47:28.082+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:47:28 audit: created new audit service 2021-08-16T13:47:28.125+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:47:28.125+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:47:28.125+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:47:28.125+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:47:28.126+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:47:28.159+00:00 [Info] security context initialized 2021-08-16T13:47:28.159+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:47:28.159+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:47:28.159+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:47:28.159+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:47:28.159+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:47:28.194+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:47:28.195+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:47:28.198+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:47:28.263+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:47:28.265+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:47:28.269+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:47:28.269+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:47:28.269+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:47:28.269+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:47:28.271+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:47:28.273+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:47:28.409+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:47:28.410+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 52 us 2021-08-16T13:47:28.410+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:47:28.411+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:47:28.412+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:47:28.412+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:47:28.412+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:47:28.412+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:47:28.412+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:47:28.412+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:47:28.416+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:28.416+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:28.428+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:47:28.428+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:47:28.428+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:47:28.428+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:28.429+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:47:28.429+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:47:28.429+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:47:28.429+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:47:28.429+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:47:28.429+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:47:28.429+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:47:28.432+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_CREATED Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 2.696693ms 2021-08-16T13:47:28.432+00:00 [Warn] Indexer::validateIndexInstMap State INDEX_STATE_CREATED Stream NIL_STREAM Deferred true Found. Updating State to Ready InstId: 633044510811627291 Defn: DefnId: 12750664089235961203 Name: def_primary Using: plasma Bucket: travel-sample Scope/Id: _default/0 Collection/Id: _default/0 IsPrimary: true NumReplica: 0 InstVersion: 0 SecExprs: ([]) Desc: [] PartitionScheme: SINGLE HashScheme: CRC32 PartitionKeys: [] WhereExpr: () RetainDeletedXATTR: false State: INDEX_STATE_CREATED RState: RebalActive Stream: NIL_STREAM Version: 0 ReplicaId: 0 PartitionContainer: &{map[0:{0 0 [:9105]}] 1024 1 1024 SINGLE 0} 2021-08-16T13:47:28.432+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [ InstId: 633044510811627291 Defn: DefnId: 12750664089235961203 Name: def_primary Using: plasma Bucket: travel-sample Scope/Id: _default/0 Collection/Id: _default/0 IsPrimary: true NumReplica: 0 InstVersion: 0 SecExprs: ([]) Desc: [] PartitionScheme: SINGLE HashScheme: CRC32 PartitionKeys: [] WhereExpr: () RetainDeletedXATTR: false State: INDEX_STATE_READY RState: RebalActive Stream: NIL_STREAM Version: 0 ReplicaId: 0 PartitionContainer: &{map[0:{0 0 [:9105]}] 1024 1 1024 SINGLE 0} ] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:47:28.432+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:47:28.433+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:47:28.433+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:28.433+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:47:28.433+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:47:28.433+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:47:28.433+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:47:28.447+00:00 [Info] lifecycleMgr.dispatchRequest: op OPCODE_UPDATE_INDEX_INST elapsed 15.108482ms len(expediates) 0 len(incomings) 0 len(outgoings) 0 len(parallels) 0 error 2021-08-16T13:47:28.468+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:47:28.468+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:47:28.468+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:47:28.468+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:47:28.468+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x4004f3025f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x4004f3025f pc=0xbd8ea4] goroutine 171 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x40044069f0 sp=0x40044069c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004406a30 sp=0x40044069f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x40000aad00, 0x400470b800, 0x11a9890, 0x4004406b40, 0x40000aad18, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004406ab0 sp=0x4004406a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x40000aad00, 0x400470b800, 0x11a9890, 0x4004406b40, 0x40000aad18, 0xffff92823b88, 0x4004406c18, 0x49) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004406af0 sp=0x4004406ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4004704500, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004406da0 sp=0x4004406af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4004704500, 0x4001421110, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004406e60 sp=0x4004406da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004c98c70, 0x4004c98c80, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004407200 sp=0x4004406e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004c98c70, 0x4004c98c80, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x40044078f0 sp=0x4004407200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004f3c170, 0x4004f2a380, 0x4004cfc580, 0x4004c98c50) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004407fb0 sp=0x40044078f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004407fb0 sp=0x4004407fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x4000154000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x4000191920, 0x4, 0x1136a87, 0xf, 0x400010be68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 18 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x40001a4108) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 19 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x40001a4120) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 20 [IO wait]: internal/poll.runtime_pollWait(0xffff92821248, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400019c818, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400019c800, 0x40001ff000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400019c800, 0x40001ff000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001940a8, 0x40001ff000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x400018e840, 0x40000fa001, 0xdff, 0xdff, 0x40000a38e8, 0x6902d8, 0x4000107948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x40001a4168, 0x40000fa001, 0xdff, 0xdff, 0x40001079f8, 0x6f1968, 0x4000107938) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x4000140000, 0xa, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4000140000, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4000140000, 0xf5a960, 0x4000100560, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4000100540, 0x40000462c0, 0x4000107b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400011c1e0, 0x12f8f98, 0x4000100540, 0x4000036be7, 0x16, 0x400000cfc0, 0x40000a3c01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400011c1e0, 0x12f8f98, 0x4000100540, 0x400011c1e0, 0x4000112080, 0x4000112090, 0x400013e000, 0x4000114040, 0xee0980, 0x400000cfc0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400011c1e0, 0x12f8f98, 0x4000100540) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4000190720, 0x40000a3f78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x40000a9f78, 0x4000190720, 0x12db8a8, 0x40001a4138, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x4000190720, 0x40001b2500, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x4000190720, 0x40001b2500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 21 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 22 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 23 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 24 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 25 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 34 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400011e000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 26 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40000557a0, 0x4, 0x4000055668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40000557a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x400008a060, 0x40000557a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x40000557a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 9 [select]: net/http.(*persistConn).writeLoop(0x4000172120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 14 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400019ae80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 8 [select]: net/http.(*persistConn).readLoop(0x4000172120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 39 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000192810, 0x12c18bc, 0x1, 0x40000a5e58, 0x4000038180, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x400017fec8, 0x4000038180, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40000a5fa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x40000816c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 40 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x40000816c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 54 [IO wait]: internal/poll.runtime_pollWait(0xffff92821160, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400013e198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400013e180, 0x40000fb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400013e180, 0x40000fb000, 0x1000, 0x1000, 0x400012da88, 0x68f7a4, 0x400139a1b0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010630, 0x40000fb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000172120, 0x40000fb000, 0x1000, 0x1000, 0xf12180, 0x400011c360, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400008a4e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400008a4e0, 0xa, 0x1008880, 0x400011c320, 0x199, 0xad9, 0x400012db00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400008a4e0, 0x2, 0x0, 0x9, 0x56b, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40003b6900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40003b6900, 0x40013aa000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400019a040, 0x40013aa000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400019a040, 0x40013aa000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400019a080, 0x40013aa000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x400139a160, 0x0, 0x400012deb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400139a160, 0x0, 0x0, 0x400012df70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400139a160, 0xee0d40, 0x400011c320, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400016e120, 0x4000378d80, 0x40001c4240) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 52 [IO wait]: internal/poll.runtime_pollWait(0xffff92821078, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f8198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f8180, 0x4001398000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f8180, 0x4001398000, 0x1000, 0x1000, 0x4000131be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010008, 0x4001398000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c7e0, 0x4001398000, 0x1000, 0x1000, 0x4000131cf8, 0x67867c, 0x4000152cc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000378c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000378c00, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000c7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 53 [select]: net/http.(*persistConn).writeLoop(0x400000c7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 85 [select]: net/http.(*persistConn).writeLoop(0x40001f6a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 84 [select]: net/http.(*persistConn).readLoop(0x40001f6a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 86 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40003ec5c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 47 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x40000f07e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 16 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x40000f07e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 45 [select]: net/http.(*persistConn).readLoop(0x400000d8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 46 [select]: net/http.(*persistConn).writeLoop(0x400000d8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 48 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x40000f07e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 49 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x40000f07e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 106 [select]: net/http.(*persistConn).writeLoop(0x400000cc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 29 [select]: net/http.(*persistConn).readLoop(0x40001f6000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 30 [select]: net/http.(*persistConn).writeLoop(0x40001f6000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 66 [IO wait]: internal/poll.runtime_pollWait(0xffff92820938, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400013e818, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400013e800, 0x400139c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400013e800, 0x400139c000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40044160a0, 0x400139c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001f6a20, 0x400139c000, 0x1000, 0x1000, 0x4004418480, 0x200000003, 0x4004418480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003feea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003feea0, 0x40013b000a, 0x40000a38c8, 0x4d3b10, 0x400008a0c0, 0x40013b0000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003feea0, 0x0, 0x1, 0x0, 0x400008a0d8, 0x40013b0000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400140d350) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400140d350, 0x400443a000, 0x1000, 0x1000, 0x40000a39b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000118640, 0x400443a000, 0x1000, 0x1000, 0x400008a9b8, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000118640, 0x400443a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000118680, 0x400443a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40000a3dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000a3dd0, 0xa, 0x400443a101, 0x1, 0xe85, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40000a3dd0, 0x400443a10a, 0x1, 0x40024051e9, 0x0, 0x400443a17b, 0x1, 0xe85, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40000a3dd0, 0x40001a710a, 0x40024051e9, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40001a6fb0, 0x4000138588, 0x17, 0x40000a3f20, 0x4000371188, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x40001a6fb0, 0x113075a, 0x7, 0x4000371188, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x40001a6f70) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 67 [IO wait]: internal/poll.runtime_pollWait(0xffff92820b08, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004430098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004430080, 0x4004434000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004430080, 0x4004434000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4004416008, 0x4004434000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001f6000, 0x4004434000, 0x1000, 0x1000, 0x4004418600, 0x200000003, 0x4004418600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003fe480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003fe480, 0x40013b000a, 0x400010b8c8, 0x4d3b10, 0x400008a0c0, 0x40013b0000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003fe480, 0x0, 0x1, 0x0, 0x400008a0d8, 0x40013b0000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400442acc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400442acc0, 0x400139e000, 0x1000, 0x1000, 0x400010b9b8, 0x0, 0x400010b988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004410700, 0x400139e000, 0x1000, 0x1000, 0x4000120900, 0x3, 0x400010bbe0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004410700, 0x400139e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004410740, 0x400139e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400010bdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400010bdd0, 0xa, 0x400139e201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400010bdd0, 0x400139e20a, 0x1, 0x4000037d5b, 0x0, 0x400139e263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400010bdd0, 0x400011440a, 0x4000037d5b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40001a6fb0, 0x400011a420, 0x24, 0x400010bf20, 0x40003e4d20, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x40001a6fb0, 0x113075a, 0x7, 0x40003e4d20, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x40001a6f70) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 76 [IO wait]: internal/poll.runtime_pollWait(0xffff928204b0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f9518, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f9500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f9500, 0x4714f4, 0x40003714a0, 0x12ed9d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004bf7bf0, 0x1, 0x12ed9d0, 0x40003714a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004bf7bf0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x4004d18080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 75 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4004411a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 68 [IO wait]: internal/poll.runtime_pollWait(0xffff92820a20, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f8298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f8280, 0x400001e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f8280, 0x400001e000, 0x1000, 0x1000, 0x40000a7688, 0x69f178, 0x40000a76f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000110120, 0x400001e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000d8c0, 0x400001e000, 0x1000, 0x1000, 0x5, 0x400243c000, 0x40000a77c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000101260) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000101260, 0x68f90a, 0x4000174000, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000101260, 0x41c300, 0xffff92824518, 0x0, 0x40000a7888, 0x41cd4c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400140d650) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400140d650, 0x4004445000, 0x1000, 0x1000, 0x40000a7998, 0x44e884, 0xffff92913f18) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000118740, 0x4004445000, 0x1000, 0x1000, 0x400008a9b8, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000118740, 0x4004445000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000118780, 0x4004445000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40000a7d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000a7d80, 0xa, 0x4004445201, 0x1, 0xffff68088cf8, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40000a7d80, 0x400444520a, 0x1, 0x400452d390, 0x0, 0x4004445259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40000a7d80, 0x400244a00a, 0x400452d390, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40001a6fb0, 0x40003e3ac0, 0x1f, 0x40000a7ee0, 0x40003e4ea0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x40001a6fb0, 0x113075a, 0x7, 0x4002405200, 0xd, 0x40003e4ea0, 0x0, 0x4714f4, 0x40001a6fb0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x40001a6f70, 0x4002405200, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 87 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x40003ff260) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 88 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x400442afc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 89 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x400013e880) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 162 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x4000174f20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 90 [IO wait]: internal/poll.runtime_pollWait(0xffff92820768, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400013e998, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400013e980, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400013e980, 0x0, 0x4000124e98, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40003714a0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40003714a0, 0x12ed9d0, 0x40003714a0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x400013e900) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 91 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400013d520, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 92 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400013d520, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 93 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x400013d520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 94 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x400442baa0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 71 [IO wait]: internal/poll.runtime_pollWait(0xffff92820680, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f8598, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f8580, 0x40003f2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f8580, 0x40003f2000, 0x1000, 0x1000, 0x400012abe0, 0x469b10, 0x4001395380) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010558, 0x40003f2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003ca6c0, 0x40003f2000, 0x1000, 0x1000, 0x400012acf8, 0x67867c, 0x4000038f00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400008af60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x400008af60, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40003ca6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 77 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x4004d184c0, 0x40001cb340, 0x40001cb3b0, 0x40045cb0d0, 0x12e8f90, 0x40000f0fc0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x40045f7c44, 0x5, 0x4004d18080, 0x12eca70, 0x4000081f00, 0x130d478, 0x4000170310, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x400039eb60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x400039eb60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 72 [select]: net/http.(*persistConn).writeLoop(0x40003ca6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 105 [select]: net/http.(*persistConn).readLoop(0x400000cc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 97 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40000e8150) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 113 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x4000174f20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 107 [IO wait]: internal/poll.runtime_pollWait(0xffff92820598, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004430898, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004430880, 0x4004452000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004430880, 0x4004452000, 0x1000, 0x1000, 0x4000130a48, 0x652ce0, 0x12edac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000110168, 0x4004452000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000cc60, 0x4004452000, 0x1000, 0x1000, 0x113674e, 0x412bc0, 0x4000094280) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003793e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003793e0, 0x100a, 0x0, 0x0, 0x0, 0x0, 0x40013bf438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003793e0, 0x0, 0x40013bf478, 0x655784, 0x40013bf4a8, 0x40b2f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400447cb10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400447cb10, 0x400445f800, 0x200, 0x200, 0x60, 0x0, 0x40013bf501) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000119280, 0x400445f800, 0x200, 0x200, 0x40001c4a20, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000119280, 0x400445f800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001192c0, 0x400445f800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x400477d340, 0x680d00, 0x40013bf6b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400477d340, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400477d340, 0xee0d40, 0x4000094960, 0x12db068, 0x40001192c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400475ad80, 0x40047783c0, 0x4004c960c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 114 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400019afc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 115 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40044111c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 78 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40000f0fc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 79 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40000f0fc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 80 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x4000396fc0, 0x4004bfe5a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 81 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x400019af80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 130 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x40013a5400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 131 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x40013a5400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 132 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x4000010b18) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 133 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x40013a54a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 134 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x40013a54a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 135 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x4004d184c0, 0x40045cb0d0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 136 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x40013a5400, 0x4004bfe6c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 55 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000192810, 0x1144559, 0x1b, 0x4004621e38, 0x400244f260, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x400010f0b0, 0x400244f260, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004621f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4002430420, 0x400010f0b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 56 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 57 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400011c640, 0xd00007, 0x400460c068, 0x5, 0x400010f0c0, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 58 [IO wait]: internal/poll.runtime_pollWait(0xffff92820850, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400019c218, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400019c200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400019c200, 0xffff92913108, 0x30, 0x4004bd7dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4001436bd0, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4001436bd0, 0x4004bd7dd0, 0x11ae500, 0x4002454160, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40024540e0, 0x12ed9d0, 0x4001436bd0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40024540e0, 0x12ed9d0, 0x4001436bd0, 0x4000154000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 59 [IO wait]: internal/poll.runtime_pollWait(0xffff928203c8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400019c318, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400019c300, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400019c300, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4001436be8, 0xffff92913108, 0xffff5296ef58, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4001436be8, 0x665e1c, 0x0, 0xfe5e60, 0x40013f2dc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4001436c00, 0x4004bd7e60, 0x11ae500, 0x4002454240, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40024541c0, 0x12ec710, 0x4001436c00, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40024541c0, 0x12ec710, 0x4001436c00, 0x4000154000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 111 [IO wait]: internal/poll.runtime_pollWait(0xffff529d4c18, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400019c798, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400019c780, 0x400464c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400019c780, 0x400464c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000194348, 0x400464c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000172d80, 0x400464c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002431800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002431800, 0xa, 0x1, 0x2, 0x1311860, 0x1, 0x40013bfc38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4002431800, 0x2, 0x40013bfc78, 0x655784, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400447cea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400447cea0, 0x400445fe00, 0x200, 0x200, 0x4, 0x0, 0x40013bfd08) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000119400, 0x400445fe00, 0x200, 0x200, 0x400003e607, 0x33, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000119400, 0x400445fe00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000119440, 0x400445fe00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x400477db80, 0x1311800, 0x40013bfeb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400477db80, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400477db80, 0xee0d40, 0x4000094a00, 0x12db068, 0x4000119440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400475afc0, 0x4004778600, 0x4004c96600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 109 [IO wait]: internal/poll.runtime_pollWait(0xffff529d4de8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004430d98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004430d80, 0x4004ca2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004430d80, 0x4004ca2000, 0x1000, 0x1000, 0x4001404be0, 0x469b10, 0x4004d6a180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000110230, 0x4004ca2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40045fd440, 0x4004ca2000, 0x1000, 0x1000, 0x4001404cf8, 0x67867c, 0x4004c965a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004778540) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004778540, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40045fd440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 112 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40003ecb80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 110 [select]: net/http.(*persistConn).writeLoop(0x40045fd440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 65 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40024542a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 64 [semacquire]: sync.runtime_Semacquire(0x4004f3c178) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004f3c170) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x4004cfc580, 0x400140c3f0, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xfffff8b13969, 0x29, 0xfffff8b139cb, 0x25, 0x4004c97380, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004d1e9e0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xfffff8b13969, 0x29, 0xfffff8b139cb, 0x25, 0x4004c97380, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004d1e9e0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004407190, 0x4004406e60, 0x400140c3f0, 0x40013b6fc0, 0x400019ae80, 0x0, 0x17f, 0x4004d74da8, 0xffff5011adc8, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x4000154000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004d1e9e0, 0xb, 0x4004d1e9d8, 0x6, 0x4004d1e9f0, 0xd, 0x4004bf8c60, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x4000154000, 0x4000370000, 0x4000102360) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x4000154000, 0x4004408d00, 0x2, 0x2, 0x4004408d10, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 146 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40024542a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 147 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40024542a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 148 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40024542a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 149 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x40003ecb40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 116 [IO wait]: internal/poll.runtime_pollWait(0xffff529d4d00, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400019c618, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400019c600, 0x4004b69000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400019c600, 0x4004b69000, 0x1000, 0x1000, 0x4001400be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40044162a0, 0x4004b69000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000172c60, 0x4004b69000, 0x1000, 0x1000, 0x4001400cf8, 0x67867c, 0x4004c241e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400455b380) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x400455b380, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4000172c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 152 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4000175080, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 153 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4000175080, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 154 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4000175080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 155 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40024542a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 156 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000192810, 0x1144559, 0x1b, 0x4004d77e38, 0x400244faa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x400010f530, 0x400244faa0, 0x2c305b2c5d312d2c, 0x2d2c305b2c5d312d) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x2d2c305b2c5d312d, 0x312d2c305b2c5d31) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004d77f80, 0x312d2c305b2c5d31, 0x5d312d2c305b2c5d) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4002430ae0, 0x400010f530) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 159 [select]: net/http.(*persistConn).readLoop(0x4000172d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 160 [select]: net/http.(*persistConn).writeLoop(0x4000172d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 117 [select]: net/http.(*persistConn).writeLoop(0x4000172c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 163 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x4000174f20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 164 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x4000174f20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 165 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004cb6a00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 166 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x4004cb6a00, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 167 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x4000154000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 168 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x4000154000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 169 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x4000154000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 170 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:47:33.671+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:47:33.671+00:00 [Info] Setting ipv6=false 2021-08-16T13:47:33.681+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:47:33.683+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:47:33.683+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:47:33.683+00:00 [Info] Setting log level to Info 2021-08-16T13:47:33.683+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:47:33.686+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:47:33 audit: created new audit service 2021-08-16T13:47:33.695+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:47:33.695+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:47:33.695+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:47:33.695+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:47:33.695+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:47:33.706+00:00 [Info] security context initialized 2021-08-16T13:47:33.706+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:47:33.706+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:47:33.706+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:47:33.706+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:47:33.706+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:47:33.728+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:47:33.729+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:47:33.729+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:47:33.750+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:47:33.750+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:47:33.753+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:47:33.754+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:47:33.755+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:47:33.758+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:47:33.761+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:47:33.761+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:47:33.803+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:47:33.803+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 54 us 2021-08-16T13:47:33.803+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:47:33.805+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:47:33.805+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:47:33.815+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:47:33.815+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:47:33.815+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:47:33.815+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:47:33.815+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:47:33.815+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:33.815+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:33.816+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:47:33.816+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:47:33.816+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:47:33.817+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:33.818+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:47:33.818+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:47:33.818+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:47:33.818+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:47:33.818+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:47:33.818+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:47:33.818+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:47:33.822+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 4.16691ms 2021-08-16T13:47:33.822+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:47:33.824+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:47:33.824+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:33.824+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:47:33.824+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:47:33.824+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:47:33.824+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... unexpected fault address 0x4004b4a6df 2021-08-16T13:47:33.835+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:47:33.835+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:47:33.835+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:47:33.835+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:47:33.835+00:00 [Info] : Recover shard shard1 from metadata completed. fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x4004b4a6df pc=0xbd8ea4] goroutine 181 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004edc9f0 sp=0x4004edc9c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004edca30 sp=0x4004edc9f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x4000165a00, 0x4004b2ec00, 0x11a9890, 0x4004edcb40, 0x4000165a18, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004edcab0 sp=0x4004edca40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x4000165a00, 0x4004b2ec00, 0x11a9890, 0x4004edcb40, 0x4000165a18, 0xffff7f6db758, 0x4004edcc18, 0x49) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004edcaf0 sp=0x4004edcab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x40003e0a00, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004edcda0 sp=0x4004edcaf0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x40003e0a00, 0x40001696c0, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004edce60 sp=0x4004edcda0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004c6c6c0, 0x4004c6c6d0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004edd200 sp=0x4004edce60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004c6c6c0, 0x4004c6c6d0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004edd8f0 sp=0x4004edd200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004ec2df0, 0x4004c63500, 0x40044cbb80, 0x4004c6c6a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004eddfb0 sp=0x4004edd8f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004eddfb0 sp=0x4004eddfb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x400016e380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x4000204f60, 0x4, 0x1136a87, 0xf, 0x40000f9e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 7 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400000e690) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 8 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400000e6a8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 9 [IO wait]: internal/poll.runtime_pollWait(0xffff7f655e88, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000126098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000126080, 0x400013d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000126080, 0x400013d000, 0x1000, 0x1000, 0x1311860, 0x47b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000114018, 0x400013d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x400010e120, 0x400015c001, 0xdff, 0xdff, 0x400009b8e8, 0x6902d8, 0x40000f5948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x4000124030, 0x400015c001, 0xdff, 0xdff, 0x40000f59f8, 0x6f1968, 0x40000f5938) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x400013a160, 0xa, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400013a160, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400013a160, 0xf5a960, 0x400010e500, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x400010e4e0, 0x40001161c0, 0x40000f5b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400014c140, 0x12f8f98, 0x400010e4e0, 0x400011c8c7, 0x16, 0x4000130480, 0x400009bc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400014c140, 0x12f8f98, 0x400010e4e0, 0x400014c140, 0x400011c0d0, 0x400011c0e0, 0x4000126100, 0x40001160e0, 0xee0980, 0x4000130480, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400014c140, 0x12f8f98, 0x400010e4e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4000097ec0, 0x400009bf78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x40000a1f78, 0x4000097ec0, 0x12db8a8, 0x4000124000, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x4000097ec0, 0x40000b1360, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x4000097ec0, 0x40000b1360) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 10 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 11 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 12 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 13 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 14 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 18 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400010c300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 15 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40000547a0, 0x4, 0x4000054668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 26 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400021c2c0, 0x12c18bc, 0x1, 0x40001bfe58, 0x400012a720, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x4000198ec8, 0x400012a720, 0x4000198ec8, 0x679704) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x16) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40001bffa0, 0x0, 0x400011a100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x400011a240) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 24 [select]: net/http.(*persistConn).readLoop(0x40003d67e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 25 [select]: net/http.(*persistConn).writeLoop(0x40003d67e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 27 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x400011a240) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 92 [IO wait]: internal/poll.runtime_pollWait(0xffff7f6552c0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001f8498, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40001f8480, 0x40045b2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40001f8480, 0x40045b2000, 0x1000, 0x1000, 0x400237cbe0, 0x469b10, 0x4000197080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001880a0, 0x40045b2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001ce360, 0x40045b2000, 0x1000, 0x1000, 0x400237ccf8, 0x67867c, 0x400012b2c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000186ea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000186ea0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40001ce360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 30 [IO wait]: internal/poll.runtime_pollWait(0xffff7f655cb8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0280, 0x400001e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0280, 0x400001e000, 0x1000, 0x1000, 0x4000066be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000103e8, 0x400001e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003d6c60, 0x400001e000, 0x1000, 0x1000, 0x4000066cf8, 0x67867c, 0x4000038a80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003a97a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40003a97a0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40003d6c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 31 [select]: net/http.(*persistConn).writeLoop(0x40003d6c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 16 [IO wait]: internal/poll.runtime_pollWait(0xffff7f655da0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0180, 0x400015d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0180, 0x400015d000, 0x1000, 0x1000, 0x40001b2a88, 0x68f7a4, 0x4000446050) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000100d0, 0x400015d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003d67e0, 0x400015d000, 0x1000, 0x1000, 0xf12180, 0x400014c220, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003a9320) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003a9320, 0xa, 0x1008880, 0x400014c1e0, 0x199, 0x1000, 0x40001b2b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003a9320, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000434030) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000434030, 0x400044c000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400043e000, 0x400044c000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400043e000, 0x400044c000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400043e040, 0x400044c000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4000446000, 0x0, 0x40001b2eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4000446000, 0x0, 0x0, 0x40001b2f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4000446000, 0xee0d40, 0x400014c1e0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400042c000, 0x40000824e0, 0x4000444000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 74 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x4004ce02a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 89 [select]: net/http.(*persistConn).writeLoop(0x40001ce240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 88 [select]: net/http.(*persistConn).readLoop(0x40001ce240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 52 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400043ea40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 40 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40000e82a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 82 [select]: net/http.(*persistConn).readLoop(0x40003d6d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 83 [select]: net/http.(*persistConn).writeLoop(0x40003d6d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 54 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x40000e8b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 119 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x4004623220) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 131 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400011afc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 57 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x40000e8b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 58 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x40000e8b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 59 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x40000e8b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 60 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400043ef40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 61 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x4000417a40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 62 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x40023d4d50) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 63 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x4000127780) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 64 [IO wait]: internal/poll.runtime_pollWait(0xffff7f655578, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000127898, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000127880, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000127880, 0x0, 0x4002375698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40023fe540, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40023fe540, 0x12ed9d0, 0x40023fe540, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x4000127800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 65 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400045bad0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 98 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400045bad0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 99 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x400045bad0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 100 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x40023d5830) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 106 [IO wait]: internal/poll.runtime_pollWait(0xffff7f6553a8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000126298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000126280, 0x400462c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000126280, 0x400462c000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000114098, 0x400462c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000db00, 0x400462c000, 0x1000, 0x1000, 0x40043fa600, 0x200000003, 0x40043fa600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400010f3e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400010f3e0, 0x400039200a, 0x40001b98c8, 0x4d3b10, 0x40000820c0, 0x4000392000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400010f3e0, 0x0, 0x1, 0x0, 0x40000820d8, 0x4000392000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40003af080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40003af080, 0x4004630000, 0x1000, 0x1000, 0x40001b99b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40044f82c0, 0x4004630000, 0x1000, 0x1000, 0x400010e598, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40044f82c0, 0x4004630000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40044f8300, 0x4004630000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40001b9dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001b9dd0, 0xa, 0x4004630201, 0x1, 0xd33, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40001b9dd0, 0x400463020a, 0x1, 0x40044e5229, 0x0, 0x40046302cd, 0x1, 0xd33, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40001b9dd0, 0x40000e5e0a, 0x40044e5229, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000e4380, 0x40001509a8, 0x17, 0x40001b9f20, 0x40044f5a58, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x40000e4380, 0x113075a, 0x7, 0x40044f5a58, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x40000e4340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 73 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x4004ce02a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 103 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40000e0540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 72 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x4004ce02a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 94 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40044f8b80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 107 [IO wait]: internal/poll.runtime_pollWait(0xffff7f655490, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001f8298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40001f8280, 0x40003aa000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40001f8280, 0x40003aa000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000188070, 0x40003aa000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001ce240, 0x40003aa000, 0x1000, 0x1000, 0x40043fac00, 0x200000003, 0x40043fac00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000186a80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000186a80, 0x400039200a, 0x40001bd8c8, 0x4d3b10, 0x40000820c0, 0x4000392000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000186a80, 0x0, 0x1, 0x0, 0x40000820d8, 0x4000392000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400246c870) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400246c870, 0x40045b1000, 0x1000, 0x1000, 0x40001bd9b8, 0x0, 0x40001bd988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400011a580, 0x40045b1000, 0x1000, 0x1000, 0x4000196a80, 0x3, 0x40001bdbe0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400011a580, 0x40045b1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400011a5c0, 0x40045b1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40001bddd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001bddd0, 0xa, 0x40045b1201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40001bddd0, 0x40045b120a, 0x1, 0x4002472b6b, 0x0, 0x40045b1263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40001bddd0, 0x40001c420a, 0x4002472b6b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000e4380, 0x400011e120, 0x24, 0x40001bdf20, 0x40044f5a40, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x40000e4380, 0x113075a, 0x7, 0x40044f5a40, 0x0, 0x400014c140, 0x400011c0d0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x40000e4340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 112 [IO wait]: internal/poll.runtime_pollWait(0xffff7f6551d8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000126698, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000126680, 0x4004640000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000126680, 0x4004640000, 0x1000, 0x1000, 0x4004629688, 0x69f178, 0x40046296f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000114118, 0x4004640000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000130120, 0x4004640000, 0x1000, 0x1000, 0x5, 0x40044b0a00, 0x40046297c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400010f9e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400010f9e0, 0x68f90a, 0x4000162b00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400010f9e0, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400246cd50) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400246cd50, 0x40045c8000, 0x1000, 0x1000, 0x4004629998, 0x44e884, 0x4004629948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400011a840, 0x40045c8000, 0x1000, 0x1000, 0x400010e598, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400011a840, 0x40045c8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400011a880, 0x40045c8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4004629d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004629d80, 0xa, 0x40045c8201, 0x1, 0xffff7f653fd8, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4004629d80, 0x40045c820a, 0x1, 0x4000183c40, 0x0, 0x40045c8259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4004629d80, 0x40000ce40a, 0x4000183c40, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000e4380, 0x40044f6ec0, 0x1f, 0x4004629ee0, 0x400463e078, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x40000e4380, 0x113075a, 0x7, 0x40044e5240, 0xd, 0x400463e078, 0x0, 0x4714f4, 0x40000e4380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x40000e4340, 0x40044e5240, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 110 [select]: net/http.(*persistConn).readLoop(0x400000db00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 111 [select]: net/http.(*persistConn).writeLoop(0x400000db00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 93 [select]: net/http.(*persistConn).writeLoop(0x40001ce360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 117 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400043e180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 115 [select]: net/http.(*persistConn).readLoop(0x4000130120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 116 [select]: net/http.(*persistConn).writeLoop(0x4000130120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 118 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x4004623220) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 132 [IO wait]: internal/poll.runtime_pollWait(0xffff7f655748, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001f8a18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40001f8a00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40001f8a00, 0x12d9d48, 0x400011b040, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004bb4630, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004bb4630, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x400011ba00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 133 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x40044f9d40, 0x40001c2d20, 0x40001c2d90, 0x400040da20, 0x12e8f90, 0x40000e82a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x40046bbcb4, 0x5, 0x400011ba00, 0x12eca70, 0x400011b880, 0x130d478, 0x4000208000, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40000e8540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40000e8540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 120 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x40044f9d40, 0x400040da20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 41 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40000e82a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 42 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x4000140120, 0x40023b4a20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 43 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x400043e100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 44 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x40023a63c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 45 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x40023a63c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 46 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x4000458410) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 47 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x40023a63c0, 0x40023b4a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 48 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400021c2c0, 0x1144559, 0x1b, 0x400462be38, 0x40023b4ba0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x40023a2f70, 0x40023b4ba0, 0x4002375698, 0xccadf0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x12ed9d0, 0x40023fe540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400462bf80, 0x40023fe540, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x400460b8c0, 0x40023a2f70) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 49 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 146 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400014cc30, 0xd00007, 0x4002442c18, 0x5, 0x40023a2f80, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 147 [IO wait]: internal/poll.runtime_pollWait(0xffff7f6550f0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1e18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f1e00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f1e00, 0xffff7f6d1108, 0x30, 0x4004ad9530) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40023f9650, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40023f9650, 0x4004ad9530, 0x11ae500, 0x4004466e80, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x4004466e00, 0x12ed9d0, 0x40023f9650, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x4004466e00, 0x12ed9d0, 0x40023f9650, 0x400016e380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 148 [IO wait]: internal/poll.runtime_pollWait(0xffff7f655008, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1f18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f1f00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f1f00, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40023f9668, 0xffff7f6d1108, 0xffff7f66df38, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40023f9668, 0x665e1c, 0x0, 0xfe5e60, 0x400461f5c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x40023f9680, 0x4004ad95c0, 0x11ae500, 0x4004466f60, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x4004466ee0, 0x12ec710, 0x40023f9680, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x4004466ee0, 0x12ec710, 0x40023f9680, 0x400016e380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 144 [IO wait]: internal/poll.runtime_pollWait(0xffff7f655660, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0400, 0x400440a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0400, 0x400440a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000104a0, 0x400440a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003d6d80, 0x400440a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003a9ce0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003a9ce0, 0xa, 0x0, 0x0, 0x0, 0x0, 0x400237ac38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003a9ce0, 0x0, 0x400237ac78, 0x655784, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004bfe720) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004bfe720, 0x4004bb1600, 0x200, 0x200, 0x4, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000081680, 0x4004bb1600, 0x200, 0x200, 0x4004be6647, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000081680, 0x4004bb1600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40000816c0, 0x4004bb1600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004dc46e0, 0x0, 0x400237aeb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004dc46e0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004dc46e0, 0xee0d40, 0x40001caeb0, 0x12db068, 0x40000816c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004bb9320, 0x40046fb5c0, 0x4004d80840) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 150 [IO wait]: internal/poll.runtime_pollWait(0xffff540b1598, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004d00098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004d00080, 0x4004d02000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004d00080, 0x4004d02000, 0x1000, 0x1000, 0x400460cbe0, 0x469b10, 0x4000103380) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000458420, 0x4004d02000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001cf680, 0x4004d02000, 0x1000, 0x1000, 0x400460ccf8, 0x67867c, 0x40023b4d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400460b9e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x400460b9e0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40001cf680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 151 [select]: net/http.(*persistConn).writeLoop(0x40001cf680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 71 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x4004ce02a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 70 [semacquire]: sync.runtime_Semacquire(0x4004ec2df8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004ec2df0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x40044cbb80, 0x4000180300, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffd6ea5969, 0x29, 0xffffd6ea59cb, 0x25, 0x4004ed4060, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004bd6a50, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffd6ea5969, 0x29, 0xffffd6ea59cb, 0x25, 0x4004ed4060, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004bd6a50, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004edd190, 0x4004edce60, 0x4000180300, 0x40000b8d80, 0x400043ea40, 0x0, 0xffff54101350, 0x88, 0xffff7f6d1130, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x400016e380, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004bd6a50, 0xb, 0x4004bd6a48, 0x6, 0x4004bd6a60, 0xd, 0x4004bafaa0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x400016e380, 0x40003a0000, 0x4000110c00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x400016e380, 0x40004257a0, 0x2, 0x2, 0x40004257b0, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 75 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x40003d5340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 167 [IO wait]: internal/poll.runtime_pollWait(0xffff540b1680, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004ca4698, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004ca4680, 0x4004e76000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004ca4680, 0x4004e76000, 0x1000, 0x1000, 0x4002445be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010770, 0x4004e76000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004e5c000, 0x4004e76000, 0x1000, 0x1000, 0x4002445cf8, 0x67867c, 0x40045a5da0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004439e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004439e00, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004e5c000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 78 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004e384d0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 79 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004e384d0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 80 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4004e384d0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 81 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x4004ce02a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 162 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400021c2c0, 0x1144559, 0x1b, 0x4004e69e38, 0x40045a5080, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x400441d340, 0x40045a5080, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004e69f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40044390e0, 0x400441d340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 168 [select]: net/http.(*persistConn).writeLoop(0x4004e5c000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 165 [select]: net/http.(*persistConn).readLoop(0x4004e5c120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 166 [select]: net/http.(*persistConn).writeLoop(0x4004e5c120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 121 [IO wait]: internal/poll.runtime_pollWait(0xffff540b14b0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004ca4798, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004ca4780, 0x4004e6c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004ca4780, 0x4004e6c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010768, 0x4004e6c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004e5c120, 0x4004e6c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004439ce0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004439ce0, 0xa, 0x0, 0x0, 0x0, 0x0, 0x400461f418) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004439ce0, 0x412bc0, 0x0, 0x0, 0x400461f4c8, 0x412bc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004ad9620) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004ad9620, 0x4004b45000, 0x200, 0x200, 0x4004ad9590, 0x30, 0x400461f578) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40044f9f00, 0x4004b45000, 0x200, 0x200, 0x0, 0xffff7f6d1108, 0x400461f568) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40044f9f00, 0x4004b45000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40044f9f40, 0x4004b45000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004b43b80, 0x12ec700, 0x400461f6b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004b43b80, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004b43b80, 0xee0d40, 0x4000190eb0, 0x12db068, 0x40044f9f40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004b46870, 0x4004b2ade0, 0x4004adb7a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 122 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40003d5380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 123 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x4004e38370) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 124 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x4004e38370) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 125 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x4004e38370) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 126 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x4004e38370) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 127 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004c68600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 128 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x4004c68600, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 129 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x400016e380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 178 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x400016e380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 179 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x400016e380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 180 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:47:39.004+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:47:39.005+00:00 [Info] Setting ipv6=false 2021-08-16T13:47:39.012+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:47:39.014+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:47:39.014+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:47:39.014+00:00 [Info] Setting log level to Info 2021-08-16T13:47:39.014+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:47:39.017+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:47:39 audit: created new audit service 2021-08-16T13:47:39.024+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:47:39.024+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:47:39.024+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:47:39.024+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:47:39.024+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:47:39.036+00:00 [Info] security context initialized 2021-08-16T13:47:39.036+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:47:39.037+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:47:39.037+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:47:39.037+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:47:39.037+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:47:39.050+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:47:39.058+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:47:39.067+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:47:39.078+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:47:39.081+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:47:39.081+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:47:39.081+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:47:39.084+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:47:39.085+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:47:39.088+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:47:39.090+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:47:39.137+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:47:39.137+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 57 us 2021-08-16T13:47:39.138+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:47:39.145+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:47:39.146+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:47:39.147+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:47:39.147+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:47:39.147+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:47:39.147+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:47:39.147+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:47:39.149+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:39.149+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:39.150+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:47:39.150+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:47:39.150+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:47:39.150+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:39.151+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:47:39.151+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:47:39.151+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:47:39.151+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:47:39.151+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:47:39.151+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:47:39.151+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:47:39.154+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 2.951207ms 2021-08-16T13:47:39.154+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:47:39.155+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:47:39.155+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:39.155+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:47:39.156+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:47:39.156+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:47:39.156+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... unexpected fault address 2021-08-16T13:47:39.166+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:47:39.166+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:47:39.166+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:47:39.166+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:47:39.166+00:00 [Info] : Recover shard shard1 from metadata completed. 0x40001db45f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x40001db45f pc=0xbd8ea4] goroutine 198 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004d529f0 sp=0x4004d529c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004d52a30 sp=0x4004d529f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x40044bed00, 0x400046b800, 0x11a9890, 0x4004d52b40, 0x40044bed18, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004d52ab0 sp=0x4004d52a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x40044bed00, 0x400046b800, 0x11a9890, 0x4004d52b40, 0x40044bed18, 0xffffb105cf60, 0x4004d52c18, 0x49) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004d52af0 sp=0x4004d52ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x40003e2640, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004d52da0 sp=0x4004d52af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x40003e2640, 0x40044a5180, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004d52e60 sp=0x4004d52da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004dd0440, 0x4004dd0450, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004d53200 sp=0x4004d52e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004dd0440, 0x4004dd0450, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004d538f0 sp=0x4004d53200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004ddc8c0, 0x4004dcea80, 0x40000d1600, 0x4004dd0420) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004d53fb0 sp=0x4004d538f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004d53fb0 sp=0x4004d53fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x4000420700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x4000204f90, 0x4, 0x1136a87, 0xf, 0x40000f9e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 7 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400000e660) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 8 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400000e678) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 9 [IO wait]: internal/poll.runtime_pollWait(0xffff8872de88, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001a0098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40001a0080, 0x40001bb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40001a0080, 0x40001bb000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000194010, 0x40001bb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x400018e120, 0x40001ba001, 0xdff, 0xdff, 0x400009b8e8, 0x6902d8, 0x400044b948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400019c030, 0x40001ba001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40001b8160, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40001b8160, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40001b8160, 0xf5a960, 0x40001144a0, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4000114480, 0x4000118160, 0x400044bb28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x4000126140, 0x12f8f98, 0x4000114480, 0x40001108b7, 0x16, 0x4000398240, 0x400009bc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x4000126140, 0x12f8f98, 0x4000114480, 0x4000126140, 0x40001100c0, 0x40001100d0, 0x400011c000, 0x4000118000, 0xee0980, 0x4000398240, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x4000126140, 0x12f8f98, 0x4000114480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4000097ef0, 0x400009bf78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x40034cbf78, 0x4000097ef0, 0x12db8a8, 0x400019c000, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x4000097ef0, 0x40000b12c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x4000097ef0, 0x40000b12c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 10 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 11 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 12 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 13 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 14 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 18 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400010c100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 15 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40000547a0, 0x4, 0x4000054668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 20 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40023f6080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 40 [select]: net/http.(*persistConn).writeLoop(0x4000399440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 37 [select]: net/http.(*persistConn).readLoop(0x40001c8120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 38 [select]: net/http.(*persistConn).writeLoop(0x40001c8120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 52 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400021c300, 0x12c18bc, 0x1, 0x400009fe58, 0x4000038c60, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x400012d6c8, 0x4000038c60, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400009ffa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x40003dc740) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 53 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x40003dc740) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 58 [IO wait]: internal/poll.runtime_pollWait(0xffff8872dda0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0100, 0x40001ce000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0100, 0x40001ce000, 0x1000, 0x1000, 0x4000135a88, 0x68f7a4, 0x40001b8730) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000100b8, 0x40001ce000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c8120, 0x40001ce000, 0x1000, 0x1000, 0xf12180, 0x40001262c0, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400018e4e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400018e4e0, 0xa, 0x1008880, 0x4000126280, 0x199, 0x1000, 0x4000135b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400018e4e0, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400010ee40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400010ee40, 0x400045c000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400011a300, 0x400045c000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400011a300, 0x400045c000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400011a340, 0x400045c000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40001b86e0, 0x0, 0x4000135eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40001b86e0, 0x0, 0x0, 0x4000135f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40001b86e0, 0xee0d40, 0x4000126280, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40001a8cf0, 0x4000114a20, 0x4000038ea0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 56 [IO wait]: internal/poll.runtime_pollWait(0xffff8872dcb8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011c198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011c180, 0x4000458000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011c180, 0x4000458000, 0x1000, 0x1000, 0x4000132be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000116180, 0x4000458000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003990e0, 0x4000458000, 0x1000, 0x1000, 0x4000132cf8, 0x67867c, 0x4000038e40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001148a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40001148a0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40003990e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 57 [select]: net/http.(*persistConn).writeLoop(0x40003990e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 65 [select]: net/http.(*persistConn).readLoop(0x40034fc240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 39 [select]: net/http.(*persistConn).readLoop(0x4000399440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 83 [IO wait]: internal/poll.runtime_pollWait(0xffff8872d578, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011d018, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011d000, 0x400450d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011d000, 0x400450d000, 0x1000, 0x1000, 0x1, 0x0, 0x40001416f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000116330, 0x400450d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40034fc240, 0x400450d000, 0x1000, 0x1000, 0x5, 0x40044d8000, 0x40001417c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004504480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004504480, 0x68f90a, 0x40001418a8, 0x412bc0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004504480, 0x0, 0x4000141868, 0x41a0a8, 0x1, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400449ab40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400449ab40, 0x40044c7000, 0x1000, 0x1000, 0x4000141998, 0x44e884, 0x40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40044a2440, 0x40044c7000, 0x1000, 0x1000, 0x4004492898, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40044a2440, 0x40044c7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40044a2480, 0x40044c7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000141d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000141d80, 0xa, 0x40044c7201, 0x1, 0xffff882af418, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000141d80, 0x40044c720a, 0x1, 0x4004526c00, 0x0, 0x40044c7259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000141d80, 0x40044d260a, 0x4004526c00, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4004484520, 0x4004465120, 0x1f, 0x4000141ee0, 0x4004466618, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x4004484520, 0x113075a, 0x7, 0x4004526650, 0xd, 0x4004466618, 0x0, 0x400011a340, 0x40001b86e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x40044844e0, 0x4004526650, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 82 [select]: net/http.(*persistConn).writeLoop(0x40034fc240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 22 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x400041c8c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 27 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4004484d00, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 66 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x400041c8c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 67 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x400041c8c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 68 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x400041c8c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 25 [select]: net/http.(*persistConn).readLoop(0x40023fc5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 26 [select]: net/http.(*persistConn).writeLoop(0x40023fc5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 69 [IO wait]: internal/poll.runtime_pollWait(0xffff8872d660, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001a0d18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40001a0d00, 0x4002507000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40001a0d00, 0x4002507000, 0x1000, 0x1000, 0x40, 0x1d, 0x40001457b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000194420, 0x4002507000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40023fc5a0, 0x4002507000, 0x1000, 0x1000, 0x4004494480, 0x200000003, 0x4004494480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40025007e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40025007e0, 0x40001e200a, 0x40001458f8, 0x412bc0, 0x40000820c0, 0x40001e2000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40025007e0, 0x0, 0x1, 0x4dd090, 0x40000820d8, 0x40001e2000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40003f8c30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40003f8c30, 0x4004531000, 0x1000, 0x1000, 0x40001459b8, 0x0, 0x4000145988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400011b440, 0x4004531000, 0x1000, 0x1000, 0x4000102900, 0x3, 0x4000145be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400011b440, 0x4004531000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400011b480, 0x4004531000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000145dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000145dd0, 0xa, 0x4004531201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000145dd0, 0x400453120a, 0x1, 0x4004526639, 0x0, 0x40045312e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000145dd0, 0x400450640a, 0x4004526639, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4004484520, 0x40044a02a0, 0x17, 0x4000145f20, 0x400448e828, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x4004484520, 0x113075a, 0x7, 0x400448e828, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x40044844e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 70 [IO wait]: internal/poll.runtime_pollWait(0xffff8872d748, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011cc98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011cc80, 0x40001e7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011cc80, 0x40001e7000, 0x1000, 0x1000, 0x40, 0x1d, 0x400009b7b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010728, 0x40001e7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000399440, 0x40001e7000, 0x1000, 0x1000, 0x4004494600, 0x200000003, 0x4004494600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400018e9c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400018e9c0, 0x40001e200a, 0x400009b8c8, 0x4d3b10, 0x40000820c0, 0x40001e2000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400018e9c0, 0x0, 0x1, 0x4dd090, 0x40000820d8, 0x40001e2000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400042db00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400042db00, 0x400444b000, 0x1000, 0x1000, 0x400009b9b8, 0x10, 0x400009ba68) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40003dcdc0, 0x400444b000, 0x1000, 0x1000, 0x4004492898, 0x58, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40003dcdc0, 0x400444b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40003dce00, 0x400444b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400009bdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400009bdd0, 0xa, 0x400444b201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400009bdd0, 0x400444b20a, 0x1, 0x400452663c, 0x0, 0x400444b263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400009bdd0, 0x400004690a, 0x400452663c, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4004484520, 0x40044865a0, 0x24, 0x400009bf20, 0x400448e810, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x4004484520, 0x113075a, 0x7, 0x400448e810, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x40044844e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 33 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x40003ddbc0, 0x4000112f50) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 94 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40000e01c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 47 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400011aa00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 73 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40023f6500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 74 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x40044931a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 75 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x400449aea0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 76 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x40044b6600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 77 [IO wait]: internal/poll.runtime_pollWait(0xffff8872d1d8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40044b6718, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40044b6700, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40044b6700, 0x0, 0x400251fe98, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40045b51b8, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40045b51b8, 0x12ed9d0, 0x40045b51b8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x40044b6680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 28 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4004484d00, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 29 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x4004484d00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 30 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x4004470210) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 32 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x4000219360) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 31 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x4000219360) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 99 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40023f61c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 167 [semacquire]: sync.runtime_Semacquire(0x4004ddc8c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004ddc8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x40000d1600, 0x400010eab0, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffdf8d6969, 0x29, 0xffffdf8d69cb, 0x25, 0x40001ff8c0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004b27d20, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffdf8d6969, 0x29, 0xffffdf8d69cb, 0x25, 0x40001ff8c0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004b27d20, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004d53190, 0x4004d52e60, 0x400010eab0, 0x40023ee240, 0x40023f6080, 0x0, 0x3f, 0x40034c6da8, 0xffff8824e930, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x4000420700, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004b27d20, 0xb, 0x4004b27d18, 0x6, 0x4004b27d30, 0xd, 0x4004b6ac20, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x4000420700, 0x40003a0000, 0x4000205fb0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x4000420700, 0x4004482ef0, 0x2, 0x2, 0x4004482f00, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 168 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40000e61c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 170 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40000e61c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 80 [select]: net/http.(*persistConn).readLoop(0x40044c2900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 81 [select]: net/http.(*persistConn).writeLoop(0x40044c2900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 100 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40023f7440) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 101 [IO wait]: internal/poll.runtime_pollWait(0xffff8872d490, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011d918, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400011d900, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400011d900, 0x58ec48, 0x40034f5b60, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004b69818, 0x4714f4, 0x40034f5b60, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004b69818, 0x4004526c00, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x400011bf00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 102 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x40003ddbc0, 0x400007f110, 0x400007f180, 0x4000112f50, 0x12e8f90, 0x40024a69a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x4004b27034, 0x5, 0x400011bf00, 0x12eca70, 0x400011bdc0, 0x130d478, 0x40000e0620, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40044b4b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40044b4b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 103 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40024a69a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 104 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40024a69a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 105 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x4002528360, 0x4004ae2d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 106 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x40023f6180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 107 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x400045b220) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 108 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x400045b220) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 109 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x4000010a68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 110 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x400045b220, 0x4004ae2d80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 111 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400021c300, 0x1144559, 0x1b, 0x40000a1e38, 0x4004ae2ea0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004b7b000, 0x4004ae2ea0, 0xf1f700, 0x400021db80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40000a1f80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4000115920, 0x4004b7b000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 112 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 113 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x40001c2f00, 0xd00007, 0x4004c8e9e8, 0x5, 0x4004b7b010, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 130 [IO wait]: internal/poll.runtime_pollWait(0xffff8872d3a8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011db18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400011db00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400011db00, 0xffffb1053108, 0x30, 0x4004b71aa0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004b69db8, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004b69db8, 0x4004b71aa0, 0x11ae500, 0x40044b4cc0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40044b4c40, 0x12ed9d0, 0x4004b69db8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40044b4c40, 0x12ed9d0, 0x4004b69db8, 0x4000420700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 131 [IO wait]: internal/poll.runtime_pollWait(0xffff8872d008, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011dc18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400011dc00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400011dc00, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004b69dd0, 0xffffb1053108, 0xffff882ae648, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004b69dd0, 0x665e1c, 0x0, 0xfe5e60, 0x40046a7dc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004b69de8, 0x4004b71b30, 0x11ae500, 0x40044b4da0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40044b4d20, 0x12ec710, 0x4004b69de8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40044b4d20, 0x12ec710, 0x4004b69de8, 0x4000420700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 132 [IO wait]: internal/poll.runtime_pollWait(0xffff8872d0f0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40044b6818, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40044b6800, 0x400015a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40044b6800, 0x400015a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40044ae0d8, 0x400015a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40044c2900, 0x400015a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40044937a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40044937a0, 0xa, 0x0, 0x0, 0x0, 0x0, 0x40046a7c18) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40044937a0, 0x412bc0, 0x0, 0x0, 0x40046a7cc8, 0x412bc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004b71b90) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004b71b90, 0x4004b2f000, 0x200, 0x200, 0x4004b71b00, 0x30, 0x40046a7d78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000081b40, 0x4004b2f000, 0x200, 0x200, 0x0, 0xffffb1053108, 0x40046a7d68) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000081b40, 0x4004b2f000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000081bc0, 0x4004b2f000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004b2b760, 0x12ec700, 0x40046a7eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004b2b760, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004b2b760, 0xee0d40, 0x40001c2ff0, 0x12db068, 0x4000081bc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004c8c750, 0x4004ca4000, 0x4004ae3020) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 163 [IO wait]: internal/poll.runtime_pollWait(0xffff88528028, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40044b6998, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40044b6980, 0x40001e0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40044b6980, 0x40001e0000, 0x1000, 0x1000, 0x4004540be0, 0x469b10, 0x4004494900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40044ae130, 0x40001e0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40023fd560, 0x40001e0000, 0x1000, 0x1000, 0x4004540cf8, 0x67867c, 0x400014f800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004493bc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004493bc0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40023fd560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 169 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40000e61c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 164 [select]: net/http.(*persistConn).writeLoop(0x40023fd560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 171 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40000e61c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 172 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x40044a2f40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 186 [IO wait]: internal/poll.runtime_pollWait(0xffff88527f40, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40044b6c18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40044b6c00, 0x4004d96000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40044b6c00, 0x4004d96000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40044ae218, 0x4004d96000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40044c2d80, 0x4004d96000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004d84c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004d84c60, 0xa, 0x0, 0x0, 0x0, 0x0, 0x4000129438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004d84c60, 0x0, 0x4000129478, 0x655784, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400010ffb0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400010ffb0, 0x40001d7000, 0x200, 0x200, 0x4, 0x0, 0x4000129508) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40044a30c0, 0x40001d7000, 0x200, 0x200, 0x40001d7000, 0x200, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40044a30c0, 0x40001d7000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40044a3100, 0x40001d7000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004d82b00, 0x0, 0x40001296b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004d82b00, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004d82b00, 0xee0d40, 0x4004490870, 0x12db068, 0x40044a3100) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004d80750, 0x4004d84f00, 0x40001ff2c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 184 [IO wait]: internal/poll.runtime_pollWait(0xffff8872d2c0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40044b6b18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40044b6b00, 0x4004d98000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40044b6b00, 0x4004d98000, 0x1000, 0x1000, 0x4004d9fbe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40044ae220, 0x4004d98000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40044c2c60, 0x4004d98000, 0x1000, 0x1000, 0x4004d9fcf8, 0x67867c, 0x40001ff260) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004d84d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004d84d80, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40044c2c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 175 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40044993f0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 176 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40044993f0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 177 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x40044993f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 178 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40000e61c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 179 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400021c300, 0x1144559, 0x1b, 0x4000147e38, 0x40001fe540, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004483bd0, 0x40001fe540, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000147f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004493f20, 0x4004483bd0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 185 [select]: net/http.(*persistConn).writeLoop(0x40044c2c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 182 [select]: net/http.(*persistConn).readLoop(0x40044c2d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 183 [select]: net/http.(*persistConn).writeLoop(0x40044c2d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 187 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40044a2f80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 188 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x4004499290) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 189 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x4004499290) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 190 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x4004499290) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 191 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x4004499290) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 192 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x40044bbe00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 193 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x40044bbe00, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 194 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x4000420700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 195 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x4000420700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 196 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x4000420700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 197 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:47:44.344+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:47:44.345+00:00 [Info] Setting ipv6=false 2021-08-16T13:47:44.352+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:47:44.358+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:47:44.358+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:47:44.358+00:00 [Info] Setting log level to Info 2021-08-16T13:47:44.358+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:47:44.361+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:47:44 audit: created new audit service 2021-08-16T13:47:44.369+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:47:44.369+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:47:44.370+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:47:44.370+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:47:44.370+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:47:44.380+00:00 [Info] security context initialized 2021-08-16T13:47:44.380+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:47:44.381+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:47:44.381+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:47:44.381+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:47:44.381+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:47:44.394+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:47:44.395+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:47:44.407+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:47:44.425+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:47:44.425+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:47:44.427+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:47:44.427+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:47:44.429+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:47:44.429+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:47:44.430+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:47:44.430+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:47:44.480+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:47:44.480+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 204 us 2021-08-16T13:47:44.481+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:47:44.482+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:47:44.483+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:47:44.486+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:47:44.486+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:47:44.486+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:47:44.486+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:47:44.486+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:47:44.486+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:44.486+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:44.488+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:47:44.488+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:47:44.488+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:47:44.488+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:44.488+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:47:44.488+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:47:44.488+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:47:44.488+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:47:44.488+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:47:44.488+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:47:44.488+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:47:44.494+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 5.204778ms 2021-08-16T13:47:44.494+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:47:44.495+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:47:44.495+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:44.495+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:47:44.495+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:47:44.495+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:47:44.495+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:47:44.506+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:47:44.506+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:47:44.506+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:47:44.506+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:47:44.506+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x40001cdd5f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x40001cdd5f pc=0xbd8ea4] goroutine 189 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004f5c9f0 sp=0x4004f5c9c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004f5ca30 sp=0x4004f5c9f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x400444f380, 0x4004e16c00, 0x11a9890, 0x4004f5cb40, 0x400444f398, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004f5cab0 sp=0x4004f5ca40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x400444f380, 0x4004e16c00, 0x11a9890, 0x4004f5cb40, 0x400444f398, 0x3f, 0x4004f5cb78, 0x412ddc) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004f5caf0 sp=0x4004f5cab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4000396640, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004f5cda0 sp=0x4004f5caf0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4000396640, 0x40013a5180, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004f5ce60 sp=0x4004f5cda0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004e64a00, 0x4004e64a10, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004f5d200 sp=0x4004f5ce60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004e64a00, 0x4004e64a10, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004f5d8f0 sp=0x4004f5d200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004f4b000, 0x40013d9500, 0x4002491080, 0x4004e649e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004f5dfb0 sp=0x4004f5d8f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004f5dfb0 sp=0x4004f5dfb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x40000aa000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x4000193dd0, 0x4, 0x1136a87, 0xf, 0x4000191e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 20 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400012a678) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 21 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400012a690) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 22 [IO wait]: internal/poll.runtime_pollWait(0xffff6d5ca388, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000188898, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000188880, 0x40001bd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000188880, 0x40001bd000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001385a0, 0x40001bd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x400013ac60, 0x40001c2001, 0xdff, 0xdff, 0x400011d8e8, 0x6902d8, 0x400018d948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400012a6d8, 0x40001c2001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40001669a0, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40001669a0, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40001669a0, 0xf5a960, 0x400013ad40, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x400013ad20, 0x4000142780, 0x400018db28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x4000100820, 0x12f8f98, 0x400013ad20, 0x4000135407, 0x16, 0x4000147440, 0x400011dc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x4000100820, 0x12f8f98, 0x400013ad20, 0x4000100820, 0x4000134c18, 0x4000134c20, 0x4000188900, 0x40001426a0, 0xee0980, 0x4000147440, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x4000100820, 0x12f8f98, 0x400013ad20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x40001923c0, 0x400011df78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400011bf78, 0x40001923c0, 0x12db8a8, 0x400012a6a8, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x40001923c0, 0x400013d2c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x40001923c0, 0x400013d2c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 24 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 25 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 26 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 27 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 28 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 5 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400000a600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 29 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40003d47a0, 0x4, 0x40003d4668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40003d47a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x400013a000, 0x40003d47a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000138000, 0x40003d47a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 13 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001ae8d0, 0x12c18bc, 0x1, 0x400011de58, 0x4000038720, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x40003d56c8, 0x4000038720, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400011dfa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4000080080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 11 [select]: net/http.(*persistConn).readLoop(0x400000c240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 12 [select]: net/http.(*persistConn).writeLoop(0x400000c240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 14 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4000080080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 37 [select]: net/http.(*persistConn).writeLoop(0x400032e240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 34 [IO wait]: internal/poll.runtime_pollWait(0xffff6d5ca1b8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000188198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000188180, 0x40001c3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000188180, 0x40001c3000, 0x1000, 0x1000, 0x40000eebe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001382e8, 0x40001c3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c6c0, 0x40001c3000, 0x1000, 0x1000, 0x40000eecf8, 0x67867c, 0x4000102660) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003596e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40003596e0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000c6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 35 [select]: net/http.(*persistConn).writeLoop(0x400000c6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 31 [IO wait]: internal/poll.runtime_pollWait(0xffff6d5ca2a0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000e6118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000e6100, 0x40000f2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000e6100, 0x40000f2000, 0x1000, 0x1000, 0x40000eaa88, 0x68f7a4, 0x40003ee050) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010038, 0x40000f2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c240, 0x40000f2000, 0x1000, 0x1000, 0xf12180, 0x40000942c0, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000092840) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000092840, 0xa, 0x1008880, 0x4000094280, 0x199, 0x1000, 0x40000eab00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000092840, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000114d50) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000114d50, 0x40003f4000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000158100, 0x40003f4000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000158100, 0x40003f4000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000158580, 0x40003f4000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40003ee000, 0x0, 0x40000eaeb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40003ee000, 0x0, 0x0, 0x40000eaf70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40003ee000, 0xee0d40, 0x4000094280, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40003da000, 0x400013a720, 0x40003ec000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 36 [select]: net/http.(*persistConn).readLoop(0x400032e240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 32 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400032ce80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 166 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x4002432e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 66 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x4002432380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 82 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x4002432380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 83 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x4002432380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 84 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x4002432380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 56 [IO wait]: internal/poll.runtime_pollWait(0xffff6d5c98a8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000e6198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000e6180, 0x40044f3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000e6180, 0x40044f3000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010090, 0x40044f3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400032e900, 0x40044f3000, 0x1000, 0x1000, 0x400009ea80, 0x200000003, 0x400009ea80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000093020) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000093020, 0x40000fc00a, 0x40000a78c8, 0x4d3b10, 0x400013a060, 0x40000fc000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000093020, 0x0, 0x1, 0x2f, 0x40000a78d8, 0x41cd4c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400253e420) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400253e420, 0x40044f6000, 0x1000, 0x1000, 0x40000a79b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400032c3c0, 0x40044f6000, 0x1000, 0x1000, 0x4000092958, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400032c3c0, 0x40044f6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400032c480, 0x40044f6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40000a7dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000a7dd0, 0xa, 0x40044f6201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40000a7dd0, 0x40044f620a, 0x1, 0x400253cf19, 0x0, 0x40044f62e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40000a7dd0, 0x400253a40a, 0x400253cf19, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000b0860, 0x400003a330, 0x17, 0x40000a7f20, 0x40013fa378, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x40000b0860, 0x113075a, 0x7, 0x40013fa378, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x40000b0820) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 81 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400032cd80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 102 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400032db80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 167 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x40000812c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 168 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40000e98c0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 85 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000159e80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 86 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x40044125a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 87 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x40044224b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 88 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x4004442400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 89 [IO wait]: internal/poll.runtime_pollWait(0xffff6d5c9b60, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004442518, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4004442500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4004442500, 0x0, 0x4004410e98, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004420690, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004420690, 0x12ed9d0, 0x4004420690, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x4004442480) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 90 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4004418680, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 91 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4004418680, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 92 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x4004418680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 93 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x4004422f90) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 94 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x4000178070) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 71 [IO wait]: internal/poll.runtime_pollWait(0xffff6d5c96d8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000188298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000188280, 0x40044fa000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000188280, 0x40044fa000, 0x1000, 0x1000, 0x400256d688, 0x69f178, 0x400256d6f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001380e0, 0x40044fa000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40023b6120, 0x40044fa000, 0x1000, 0x1000, 0x5, 0x400241c000, 0x400256d7c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40023b2660) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40023b2660, 0x68f90a, 0x40000e86e0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40023b2660, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400253eb10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400253eb10, 0x40000b2000, 0x1000, 0x1000, 0x400256d998, 0x44e884, 0xffff97271a68) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400032cac0, 0x40000b2000, 0x1000, 0x1000, 0x4000092958, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400032cac0, 0x40000b2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400032cb00, 0x40000b2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400256dd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400256dd80, 0xa, 0x40000b2201, 0x1, 0xffff6c026c98, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400256dd80, 0x40000b220a, 0x1, 0x40023b8820, 0x0, 0x40000b2259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400256dd80, 0x40000e3c0a, 0x40023b8820, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000b0860, 0x4002565e60, 0x1f, 0x400256dee0, 0x4002566450, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x40000b0860, 0x113075a, 0x7, 0x400253cf30, 0xd, 0x4002566450, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x40000b0820, 0x400253cf30, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 69 [select]: net/http.(*persistConn).readLoop(0x40001470e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 70 [select]: net/http.(*persistConn).writeLoop(0x40001470e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 57 [IO wait]: internal/poll.runtime_pollWait(0xffff6d5c9990, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4002486b18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4002486b00, 0x40044e7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4002486b00, 0x40044e7000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40003fe1d8, 0x40044e7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001470e0, 0x40044e7000, 0x1000, 0x1000, 0x400009ef00, 0x200000003, 0x400009ef00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002485080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002485080, 0x40000fc00a, 0x40000a58c8, 0x4d3b10, 0x400013a060, 0x40000fc000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4002485080, 0x0, 0x1, 0x1, 0x40000a58d8, 0x41cd4c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400007e030) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400007e030, 0x40001c7000, 0x1000, 0x1000, 0x40000a59b8, 0x0, 0x40000a5988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40024c8800, 0x40001c7000, 0x1000, 0x1000, 0x4001396480, 0x3, 0x40000a5be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40024c8800, 0x40001c7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40024c8840, 0x40001c7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40000a5dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000a5dd0, 0xa, 0x40001c7201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40000a5dd0, 0x40001c720a, 0x1, 0x40044daa8b, 0x0, 0x40001c7263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40000a5dd0, 0x400004600a, 0x40044daa8b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000b0860, 0x4000042360, 0x24, 0x40000a5f20, 0x40013fa360, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x40000b0860, 0x113075a, 0x7, 0x40013fa360, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x40000b0820) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 164 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x4002432e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 60 [select]: net/http.(*persistConn).readLoop(0x400032e900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 61 [select]: net/http.(*persistConn).writeLoop(0x400032e900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 76 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40024c89c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 74 [select]: net/http.(*persistConn).readLoop(0x40023b6120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 75 [select]: net/http.(*persistConn).writeLoop(0x40023b6120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 100 [IO wait]: internal/poll.runtime_pollWait(0xffff6d5c97c0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000189998, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000189980, 0x40045f7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000189980, 0x40045f7000, 0x1000, 0x1000, 0x400257abe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000138868, 0x40045f7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40023b7680, 0x40045f7000, 0x1000, 0x1000, 0x400257acf8, 0x67867c, 0x4004b54480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40045f8420) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40045f8420, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40023b7680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 162 [semacquire]: sync.runtime_Semacquire(0x4004f4b008) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004f4b000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x4002491080, 0x4000384ed0, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffed57c969, 0x29, 0xffffed57c9cb, 0x25, 0x4004e614a0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004b95fa0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffed57c969, 0x29, 0xffffed57c9cb, 0x25, 0x4004e614a0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004b95fa0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004f5d190, 0x4004f5ce60, 0x4000384ed0, 0x400234cb40, 0x400032ce80, 0x0, 0xbf, 0x40000a0da8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x40000aa000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004b95fa0, 0xb, 0x4004b95f98, 0x6, 0x4004b95fb0, 0xd, 0x4004bb3600, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x40000aa000, 0x4000350000, 0x40000a8030) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x40000aa000, 0x4004416530, 0x2, 0x2, 0x4004416540, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 163 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x4002432e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 101 [select]: net/http.(*persistConn).writeLoop(0x40023b7680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 103 [IO wait]: internal/poll.runtime_pollWait(0xffff6d5c9a78, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004bbc218, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4004bbc200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4004bbc200, 0x12d9d48, 0x400032dc40, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004bb5278, 0x0, 0x40003ec1e0, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004bb5278, 0x400253da00, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x4004be00c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 104 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x400442f380, 0x400019d1f0, 0x400019d260, 0x4004417fa0, 0x12e8f90, 0x4004440700, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x4004b95014, 0x5, 0x4004be00c0, 0x12eca70, 0x4000159c80, 0x130d478, 0x40000e4310, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x4004440d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x4004440d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 95 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x400441cbe0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 96 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x400441cbe0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 97 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x400442f380, 0x4004417fa0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 114 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x4004440700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 115 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x4004440700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 116 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x40044e0bd0, 0x400456c660) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 117 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x40024c8980) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 118 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x400441cc80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 119 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x400441cc80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 120 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x400443a318) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 121 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x400441cc80, 0x400456c6c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 122 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001ae8d0, 0x1144559, 0x1b, 0x4004779e38, 0x400456c7e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004760ac0, 0x400456c7e0, 0xf1f700, 0x40003ce150) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004779f80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x400476e4e0, 0x4004760ac0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 123 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 124 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400444a8c0, 0xd00007, 0x400456b958, 0x5, 0x4004760ad0, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 125 [IO wait]: internal/poll.runtime_pollWait(0xffff6d5c95f0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004442b98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4004442b80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4004442b80, 0xffff97271f18, 0x30, 0x400472f020) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x400472c7c8, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x400472c7c8, 0x400472f020, 0x11ae500, 0x400235a860, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x400235a7e0, 0x12ed9d0, 0x400472c7c8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x400235a7e0, 0x12ed9d0, 0x400472c7c8, 0x40000aa000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 126 [IO wait]: internal/poll.runtime_pollWait(0xffff6d5c9508, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004442c98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4004442c80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4004442c80, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x400472c7e0, 0xffff97271f18, 0xffff575b0820, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x400472c7e0, 0x665e1c, 0x0, 0xfe5e60, 0x4004759dc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x400472c7f8, 0x400472f0b0, 0x11ae500, 0x400235a940, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x400235a8c0, 0x12ec710, 0x400472c7f8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x400235a8c0, 0x12ec710, 0x400472c7f8, 0x40000aa000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 133 [IO wait]: internal/poll.runtime_pollWait(0xffff6d5c9c48, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000e6c18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000e6c00, 0x40001bf000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000e6c00, 0x40001bf000, 0x1000, 0x1000, 0x4004bd9ad8, 0x412bc0, 0x4004bd9a38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001387f0, 0x40001bf000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400032e240, 0x40001bf000, 0x1000, 0x1000, 0xffff572ef328, 0x0, 0x14) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003599e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003599e0, 0xa, 0x40001c0123, 0x11320c7, 0x9, 0x1, 0x4004759c9b) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003599e0, 0xd4fa28, 0x4000132200, 0x46, 0x11320c7, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400472fe60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400472fe60, 0x4004d18600, 0x200, 0x200, 0x400456bce0, 0x400456bce0, 0x400442a810) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400442fb40, 0x4004d18600, 0x200, 0x200, 0x0, 0xffff97271f18, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400442fb40, 0x4004d18600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400442fb80, 0x4004d18600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004571ce0, 0x12ec700, 0x4004759eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004571ce0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004571ce0, 0xee0d40, 0x400444af50, 0x12db068, 0x400442fb80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4002345950, 0x400476f260, 0x400456d2c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 106 [IO wait]: internal/poll.runtime_pollWait(0xffff57309d80, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004bbc418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004bbc400, 0x4004d80000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004bbc400, 0x4004d80000, 0x1000, 0x1000, 0x4004bdabe0, 0x469b10, 0x400476d080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000138c08, 0x4004d80000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40044537a0, 0x4004d80000, 0x1000, 0x1000, 0x4004bdacf8, 0x67867c, 0x4004b557a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40045f9200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40045f9200, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40044537a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 107 [select]: net/http.(*persistConn).writeLoop(0x40044537a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 165 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x4002432e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 169 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40000e98c0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 170 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x40000e98c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 171 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x4002432e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 172 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001ae8d0, 0x1144559, 0x1b, 0x40000a3e38, 0x4004e600c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004e64050, 0x4004e600c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40000a3f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004e42300, 0x4004e64050) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 177 [IO wait]: internal/poll.runtime_pollWait(0xffff57309e68, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000e7718, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000e7700, 0x40013e3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000e7700, 0x40013e3000, 0x1000, 0x1000, 0x4002512a48, 0x4000107b00, 0x4000107b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000102b8, 0x40013e3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400032f9e0, 0x40013e3000, 0x1000, 0x1000, 0x0, 0x4000150dc0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004e42ba0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004e42ba0, 0xa, 0x0, 0x0, 0x0, 0x0, 0x400440dc38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004e42ba0, 0x0, 0x400440dc78, 0x655784, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004e4ae10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004e4ae10, 0x4004e4c600, 0x200, 0x200, 0x4, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000081400, 0x4004e4c600, 0x200, 0x200, 0x4000035c07, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000081400, 0x4004e4c600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000081440, 0x4004e4c600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004e49760, 0x0, 0x400440deb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004e49760, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004e49760, 0xee0d40, 0x4000344aa0, 0x12db068, 0x4000081440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004e623f0, 0x4004e42d20, 0x4004e60c00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 175 [select]: net/http.(*persistConn).readLoop(0x400032f9e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 176 [select]: net/http.(*persistConn).writeLoop(0x400032f9e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 178 [runnable]: net.IP.IsLoopback(0x0, 0x0, 0x0, 0x10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/ip.go:121 +0x130 github.com/couchbase/cbauth/cbauthimpl.matchHost(0x4000154360, 0x2e, 0x40001352e8, 0x6, 0x4000136780, 0x20, 0x40001640e0, 0x18, 0x1c, 0x1, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:114 +0x90 github.com/couchbase/cbauth/cbauthimpl.getMemcachedCreds(0x4000154360, 0x2e, 0x40001352e8, 0x6, 0x4000136780, 0x20, 0x40001640e0, 0x18, 0x1c, 0x1, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:129 +0x4c github.com/couchbase/cbauth/cbauthimpl.GetCreds(0x400013d2c0, 0x4004c32c07, 0x9, 0x1f9b, 0x1f9b, 0x0, 0x0, 0xe, 0x0, 0x4004c32c00, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:791 +0xc0 github.com/couchbase/cbauth.(*authImpl).GetHTTPServiceAuth(0x4000138530, 0x4004c32c07, 0xe, 0x4004ea05a0, 0x15, 0x0, 0x400256f268, 0x7ff1dc, 0x6) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauth.go:188 +0x68 github.com/couchbase/cbauth.SetRequestAuthVia.func1(0x1303cf8, 0x4000138530, 0x400256f248, 0x4004c32c00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/convenience.go:31 +0x4c github.com/couchbase/cbauth.WithAuthenticator(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:136 github.com/couchbase/cbauth.SetRequestAuthVia(0x4004ea4300, 0x0, 0x0, 0x3, 0x4004c32c00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/convenience.go:30 +0x54 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004c32bd0, 0x1b, 0x400256f580, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:537 +0x13c github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004ea0480, 0x112f905, 0x6, 0x12db908, 0x4000047080, 0xee40c0, 0x4002487498, 0x400256f580, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x4004478280, 0x46, 0x12db908, 0x4000047080, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x4004478280, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x20, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400256fd40, 0x4, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x40001ccfc0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x40001ccfc0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000081300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1267 +0xf8 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 179 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x40000e9760) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 180 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x40000e9760) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 181 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x40000e9760) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 182 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x40000e9760) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 183 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004e47100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:249 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 184 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x4004e47100, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 185 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x40000aa000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 186 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x40000aa000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 187 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x40000aa000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 188 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:47:49.667+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:47:49.668+00:00 [Info] Setting ipv6=false 2021-08-16T13:47:49.681+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:47:49.683+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:47:49.683+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:47:49.683+00:00 [Info] Setting log level to Info 2021-08-16T13:47:49.683+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:47:49.685+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:47:49 audit: created new audit service 2021-08-16T13:47:49.694+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:47:49.694+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:47:49.695+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:47:49.695+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:47:49.695+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:47:49.707+00:00 [Info] security context initialized 2021-08-16T13:47:49.707+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:47:49.707+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:47:49.708+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:47:49.708+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:47:49.708+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:47:49.721+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:47:49.723+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:47:49.725+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:47:49.747+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:47:49.747+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:47:49.747+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:47:49.748+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:47:49.749+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:47:49.750+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:47:49.750+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:47:49.752+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:47:49.800+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 52 us 2021-08-16T13:47:49.800+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:47:49.799+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:47:49.801+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:47:49.802+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:47:49.806+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:47:49.806+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:47:49.806+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:47:49.806+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:47:49.806+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:47:49.806+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:49.806+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:49.808+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:47:49.808+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:47:49.808+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:47:49.808+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:49.808+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:47:49.808+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:47:49.808+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:47:49.808+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:47:49.808+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:47:49.808+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:47:49.808+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:47:49.813+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 5.023069ms 2021-08-16T13:47:49.814+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:47:49.815+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:47:49.816+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:49.816+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:47:49.816+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:47:49.816+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:47:49.816+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:47:49.826+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:47:49.826+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:47:49.826+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:47:49.827+00:00 [Info] RebalanceMgr::initService Init 2021-08-16T13:47:49.827+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:47:49.827+00:00 [Info] ServiceMgr::registerWithServer nodeuuid 8a95cde683482bdec453edbb8b18eb5d unexpected fault address 0x400476345f fatal error: fault 2021-08-16T13:47:49.827+00:00 [Info] : Recover shard shard1 from metadata completed. [signal SIGBUS: bus error code=0x1 addr=0x400476345f pc=0xbd8ea4] goroutine 195 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004d229f0 sp=0x4004d229c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004d22a30 sp=0x4004d229f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x4004770b60, 0x4004701200, 0x11a9890, 0x4004d22b40, 0x4004770b78, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004d22ab0 sp=0x4004d22a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x4004770b60, 0x4004701200, 0x11a9890, 0x4004d22b40, 0x4004770b78, 0xffff85fb5a90, 0x4004d22b78, 0x412ddc) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004d22af0 sp=0x4004d22ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4000032dc0, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004d22da0 sp=0x4004d22af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4000032dc0, 0x40001a5b20, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004d22e60 sp=0x4004d22da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004731f10, 0x4004731f20, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004d23200 sp=0x4004d22e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004731f10, 0x4004731f20, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004d238f0 sp=0x4004d23200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004dba4e0, 0x4004dc6000, 0x4000171b80, 0x4004731ef0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004d23fb0 sp=0x4004d238f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004d23fb0 sp=0x4004d23fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x40000c0380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x400019b5c0, 0x4, 0x1136a87, 0xf, 0x4000199e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 20 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x4000130678) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 21 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x4000130690) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 22 [IO wait]: internal/poll.runtime_pollWait(0xffff85e54438, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000190898, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000190880, 0x40001c5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000190880, 0x40001c5000, 0x1000, 0x1000, 0x1311860, 0x47b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400013e5a0, 0x40001c5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x4000140c60, 0x40000a6001, 0xdff, 0xdff, 0x40001238e8, 0x6902d8, 0x4000195948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x40001306d8, 0x40000a6001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40000a0000, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40000a0000, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40000a0000, 0xf5a960, 0x40000824a0, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4000082480, 0x4000046140, 0x4000195b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400008c230, 0x12f8f98, 0x4000082480, 0x40000368a7, 0x16, 0x400000c240, 0x4000123c01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400008c230, 0x12f8f98, 0x4000082480, 0x400008c230, 0x40000360b0, 0x40000360c0, 0x400009e000, 0x4000046060, 0xee0980, 0x400000c240, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400008c230, 0x12f8f98, 0x4000082480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x400019a3c0, 0x4000123f78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x4000121f78, 0x400019a3c0, 0x12db8a8, 0x40001306a8, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x400019a3c0, 0x40001432c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x400019a3c0, 0x40001432c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 23 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 24 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 25 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 26 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 27 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 5 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400000a600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 28 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40003c87a0, 0x4, 0x40003c8668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40003c87a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000140000, 0x40003c87a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x400013e000, 0x40003c87a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 14 [IO wait]: internal/poll.runtime_pollWait(0xffff85e53c10, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004422318, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004422300, 0x4004559000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004422300, 0x4004559000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400441e0b0, 0x4004559000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400442a900, 0x4004559000, 0x1000, 0x1000, 0x40000f8000, 0x200000003, 0x40000f8000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000351ec0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000351ec0, 0x40000ac00a, 0x40023578c8, 0x4d3b10, 0x4000140060, 0x40000ac000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000351ec0, 0x0, 0x1, 0x14, 0x40023578d8, 0x41cd4c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40013d0060) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40013d0060, 0x40003ee000, 0x1000, 0x1000, 0x40023579b8, 0x0, 0x4002357988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4002476000, 0x40003ee000, 0x1000, 0x1000, 0x4001396480, 0x3, 0x4002357be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4002476000, 0x40003ee000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4002476040, 0x40003ee000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4002357dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002357dd0, 0xa, 0x40003ee201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4002357dd0, 0x40003ee20a, 0x1, 0x40045548f9, 0x0, 0x40003ee2e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4002357dd0, 0x400009c50a, 0x40045548f9, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400442ec70, 0x4004414630, 0x17, 0x4002357f20, 0x400454df68, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x400442ec70, 0x113075a, 0x7, 0x400454df68, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x400442ec30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 12 [select]: net/http.(*persistConn).readLoop(0x400000cb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 13 [select]: net/http.(*persistConn).writeLoop(0x400000cb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 34 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001b6440, 0x12c18bc, 0x1, 0x40013b5e58, 0x4000038720, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x400001dec8, 0x4000038720, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40013b5fa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x40000801c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 35 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x40000801c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 38 [IO wait]: internal/poll.runtime_pollWait(0xffff85e54268, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000190098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000190080, 0x4000096000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000190080, 0x4000096000, 0x1000, 0x1000, 0x40013a4be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400013e310, 0x4000096000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400033ea20, 0x4000096000, 0x1000, 0x1000, 0x40013a4cf8, 0x67867c, 0x400001e360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000082cc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000082cc0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400033ea20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 39 [select]: net/http.(*persistConn).writeLoop(0x400033ea20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 50 [IO wait]: internal/poll.runtime_pollWait(0xffff85e54350, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400009e218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400009e200, 0x40000a7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400009e200, 0x40000a7000, 0x1000, 0x1000, 0x40013a8a88, 0x68f7a4, 0x40000a05d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010198, 0x40000a7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000cb40, 0x40000a7000, 0x1000, 0x1000, 0xf12180, 0x400008c3b0, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000082a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000082a20, 0xa, 0x1008880, 0x400008c370, 0x199, 0x1000, 0x40013a8b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000082a20, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400011b1d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400011b1d0, 0x40013bc000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000160580, 0x40013bc000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000160580, 0x40013bc000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000161580, 0x40013bc000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40000a0580, 0x0, 0x40013a8eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40000a0580, 0x0, 0x0, 0x40013a8f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40000a0580, 0xee0d40, 0x400008c370, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40000a8090, 0x4000140900, 0x4000038ae0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 123 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x400009abe0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 114 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001fae80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 51 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000161c80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 70 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001fb9c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 42 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x40001882a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 215 [IO wait]: internal/poll.runtime_pollWait(0xffff4dfdede0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400009ed98, 0x72, 0x0, 0x1, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400009ed80, 0x4004e10971, 0x1, 0x1, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400009ed80, 0x4004e10971, 0x1, 0x1, 0x40023d4601, 0x0, 0x40003586e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40003f22e8, 0x4004e10971, 0x1, 0x1, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*connReader).backgroundRead(0x4004e10960) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:692 +0x50 created by net/http.(*connReader).startBackgroundRead /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:688 +0xc4 goroutine 48 [select]: net/http.(*persistConn).readLoop(0x40024ba480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 45 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x40001882a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 46 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x40001882a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 47 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x40001882a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 49 [select]: net/http.(*persistConn).writeLoop(0x40024ba480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 15 [IO wait]: internal/poll.runtime_pollWait(0xffff85e53b28, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000191598, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000191580, 0x40024f6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000191580, 0x40024f6000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400013e950, 0x40024f6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40024ba480, 0x40024f6000, 0x1000, 0x1000, 0x4000102f00, 0x200000003, 0x4000102f00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002498e40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002498e40, 0x40000ac00a, 0x40023558c8, 0x4d3b10, 0x4000140060, 0x40000ac000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4002498e40, 0x0, 0x1, 0x0, 0x4000140078, 0x40000ac000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400033a5d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400033a5d0, 0x40044e4000, 0x1000, 0x1000, 0x40023559b8, 0x44d8a8, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40003de200, 0x40044e4000, 0x1000, 0x1000, 0x4000351a98, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40003de200, 0x40044e4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40003de240, 0x40044e4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4002355dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002355dd0, 0xa, 0x40044e4201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4002355dd0, 0x40044e420a, 0x1, 0x40044b157b, 0x0, 0x40044e4263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4002355dd0, 0x40003e210a, 0x40044b157b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400442ec70, 0x4004410a80, 0x24, 0x4002355f20, 0x400454df50, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x400442ec70, 0x113075a, 0x7, 0x400454df50, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x400442ec30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 85 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40000812c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 83 [select]: net/http.(*persistConn).readLoop(0x400442a900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 84 [select]: net/http.(*persistConn).writeLoop(0x400442a900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 86 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x4004566120) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 87 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x4000391710) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 88 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x4004422380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 89 [IO wait]: internal/poll.runtime_pollWait(0xffff85e53a40, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004422498, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4004422480, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4004422480, 0x0, 0x40003ca698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004568228, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004568228, 0x12ed9d0, 0x4004568228, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x4004422400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 90 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400442f040, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 91 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400442f040, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 92 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x400442f040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 93 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x4004570210) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 94 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40000f6070) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 53 [IO wait]: internal/poll.runtime_pollWait(0xffff85e53870, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4002466298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4002466280, 0x40013bd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4002466280, 0x40013bd000, 0x1000, 0x1000, 0x40013af688, 0x69f178, 0x40013af6f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010068, 0x40013bd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400014e5a0, 0x40013bd000, 0x1000, 0x1000, 0x5, 0x4004460500, 0x40013af7c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004566ae0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004566ae0, 0x68f90a, 0x40003d4a50, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004566ae0, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400033aed0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400033aed0, 0x40044e5000, 0x1000, 0x1000, 0x40013af998, 0x44e884, 0x40013af948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40003de900, 0x40044e5000, 0x1000, 0x1000, 0x4000351a98, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40003de900, 0x40044e5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40003de940, 0x40044e5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40013afd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40013afd80, 0xa, 0x40044e5201, 0x1, 0xffff85fbe658, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40013afd80, 0x40044e520a, 0x1, 0x4000037260, 0x0, 0x40044e5259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40013afd80, 0x40024f3a0a, 0x4000037260, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400442ec70, 0x400454e9c0, 0x1f, 0x40013afee0, 0x4004568540, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x400442ec70, 0x113075a, 0x7, 0x4004554910, 0xd, 0x4004568540, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x400442ec30, 0x4004554910, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 165 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x4002386b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 167 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x4002386b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 166 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x4002386b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 58 [select]: net/http.(*persistConn).readLoop(0x400014e5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 59 [select]: net/http.(*persistConn).writeLoop(0x400014e5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 106 [select]: net/http.(*persistConn).readLoop(0x40024ba5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 96 [IO wait]: internal/poll.runtime_pollWait(0xffff85e53788, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000191b98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000191b80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000191b80, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004bb6420, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004bb6420, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x4002476e80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 95 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40003df000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 107 [select]: net/http.(*persistConn).writeLoop(0x40024ba5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 211 [select]: net/http.(*persistConn).writeLoop(0x40024bb8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 112 [IO wait]: internal/poll.runtime_pollWait(0xffff85e536a0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000191a98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000191a80, 0x40045ca000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000191a80, 0x40045ca000, 0x1000, 0x1000, 0x4004576a48, 0x652ce0, 0x12edac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40003f2048, 0x40045ca000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40024ba5a0, 0x40045ca000, 0x1000, 0x1000, 0x113674e, 0x412bc0, 0x400008c460) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002499740) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002499740, 0x100a, 0x0, 0x0, 0x0, 0x0, 0x400001a438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4002499740, 0x0, 0x400001a478, 0x655784, 0x400001a4a8, 0x40b2f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40045b1c20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40045b1c20, 0x40045fae00, 0x200, 0x200, 0x4, 0x0, 0x400001a508) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000080f00, 0x40045fae00, 0x200, 0x200, 0x40000395c7, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000080f00, 0x40045fae00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000080f40, 0x40045fae00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40023d4580, 0x400456f800, 0x400001a6b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40023d4580, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40023d4580, 0xee0d40, 0x4000358690, 0x12db068, 0x4000080f40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400239a750, 0x4000083140, 0x40045f6b40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 97 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x4004734140, 0x40013b8a10, 0x40013b8a80, 0x4004730240, 0x12e8f90, 0x400240e460, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x40000e1dd4, 0x5, 0x4002476e80, 0x12eca70, 0x4002476d40, 0x130d478, 0x40000f60e0, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40023868c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40023868c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 121 [IO wait]: internal/poll.runtime_pollWait(0xffff85e53958, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4002467498, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4002467480, 0x400241b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4002467480, 0x400241b000, 0x1000, 0x1000, 0x4004573be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010350, 0x400241b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400033e240, 0x400241b000, 0x1000, 0x1000, 0x4004573cf8, 0x67867c, 0x40013eea80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40013f06c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40013f06c0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400033e240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 122 [select]: net/http.(*persistConn).writeLoop(0x400033e240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 124 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x400009abe0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 125 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x4004734140, 0x4004730240) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 126 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x400240e460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 127 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x400240e460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 128 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x40000fbc20, 0x40013efc20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 129 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x40001fae40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 130 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x400009ac80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 131 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x400009ac80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 132 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x4000010550) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 133 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x400009ac80, 0x40013efc80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 134 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001b6440, 0x1144559, 0x1b, 0x4002353e38, 0x40013efda0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004730d60, 0x40013efda0, 0xf1f700, 0x40001b7cc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4002353f80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40013f19e0, 0x4004730d60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 135 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 136 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400008d180, 0xd00007, 0x4004717968, 0x5, 0x4004730d70, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 137 [IO wait]: internal/poll.runtime_pollWait(0xffff85e53cf8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4002467c18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4002467c00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4002467c00, 0x0, 0x40013ead88, 0x665fe4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x400476c1f8, 0x611a6c85, 0x40013eadb8, 0x445c24) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x400476c1f8, 0x40013eae20, 0x40013eae28, 0x18, 0x4002348f00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x400447cc40, 0x12ed9d0, 0x400476c1f8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x400447cc40, 0x12ed9d0, 0x400476c1f8, 0x40000c0380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 138 [IO wait]: internal/poll.runtime_pollWait(0xffff85e535b8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4002467d18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4002467d00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4002467d00, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x400476c210, 0xffff85fb5a68, 0xffff4dfc26f8, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x400476c210, 0x665e1c, 0x0, 0xfe5e60, 0x40045b65c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x400476c228, 0x400472b5c0, 0x11ae500, 0x400447cda0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x400447cd20, 0x12ec710, 0x400476c228, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x400447cd20, 0x12ec710, 0x400476c228, 0x40000c0380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 153 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004cf08f0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 164 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x4002386b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 145 [IO wait]: internal/poll.runtime_pollWait(0xffff4dfdf268, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004d72098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004d72080, 0x4004d3d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004d72080, 0x4004d3d000, 0x1000, 0x1000, 0x40013e4be0, 0x469b10, 0x4002349500) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010600, 0x4004d3d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400476e480, 0x4004d3d000, 0x1000, 0x1000, 0x40013e4cf8, 0x67867c, 0x4004cea7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004cf27e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004cf27e0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400476e480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 163 [semacquire]: sync.runtime_Semacquire(0x4004dba4e8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004dba4e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x4000171b80, 0x4000098300, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffda80d969, 0x29, 0xffffda80d9cb, 0x25, 0x4004cebce0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004bd8ac0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffda80d969, 0x29, 0xffffda80d9cb, 0x25, 0x4004cebce0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004bd8ac0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004d23190, 0x4004d22e60, 0x4000098300, 0x4000153440, 0x4000161c80, 0x0, 0x7f, 0x40044e8da8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x40000c0380, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004bd8ac0, 0xb, 0x4004bd8ab8, 0x6, 0x4004bd8ad0, 0xd, 0x4004bb2e80, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x40000c0380, 0x4000350000, 0x4000098840) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x40000c0380, 0x4004550bd0, 0x2, 0x2, 0x4004550be0, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 146 [select]: net/http.(*persistConn).writeLoop(0x400476e480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 168 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4002477240) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 154 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004cf08f0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 155 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4004cf08f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 156 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x4002386b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 157 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001b6440, 0x1144559, 0x1b, 0x40013b3e38, 0x4004bbc840, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004731d70, 0x4004bbc840, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40013b3f80, 0x12edba8, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4002465560, 0x4004731d70) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 194 [sleep]: time.Sleep(0x12a05f200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/plasma.RunMemQuotaTunerIteration(0x40045bb7b0, 0x0, 0x400014a9b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:211 +0x24c github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:247 +0xc0 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 goroutine 160 [select]: net/http.(*persistConn).readLoop(0x400476ea20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 161 [select]: net/http.(*persistConn).writeLoop(0x400476ea20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 169 [IO wait]: internal/poll.runtime_pollWait(0xffff4dfdf180, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004d72318, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004d72300, 0x4004dbc000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004d72300, 0x4004dbc000, 0x1000, 0x1000, 0x40013a2a48, 0x652ce0, 0x12edac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010748, 0x4004dbc000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400476ea20, 0x4004dbc000, 0x1000, 0x1000, 0x113674e, 0x412bc0, 0x400008c460) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004cf3920) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004cf3920, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004cf3920, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004bb95c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004bb95c0, 0x40000dfc00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4002477300, 0x40000dfc00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4002477300, 0x40000dfc00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4002477340, 0x40000dfc00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004b7ba20, 0x0, 0x40045bd6b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004b7ba20, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004b7ba20, 0xee0d40, 0x400014ae60, 0x12db068, 0x4002477340) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004bbab40, 0x40024657a0, 0x4004bbce40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 170 [select]: net/http.(*persistConn).roundTrip(0x400033e240, 0x4002477700, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004eaa600, 0xf8, 0x10a8660, 0x4004cf8e18) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004eaa600, 0x1b6e900, 0xc03eb91f7144c414, 0x1bfa71bc29) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004eaa500, 0x12db048, 0x1b6e900, 0xc03eb91f7144c414, 0x1bfa71bc29, 0x1be4a80, 0x0, 0x400013e9b0, 0x4004bb9bf0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004bb9bc0, 0x4004eaa500, 0xc03eb91f7144c414, 0x1bfa71bc29, 0x1be4a80, 0x400013e9b0, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004bb9bc0, 0x4004eaa500, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4000043cb0, 0x1b, 0x4004cf9580, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004bbb290, 0x112f905, 0x6, 0x12db908, 0x4000047000, 0xee40c0, 0x4004eae398, 0x4004cf9580, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x4004bc0200, 0x46, 0x12db908, 0x4000047000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x4004bc0200, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x20, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004cf9d40, 0x4, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x40000ab200, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x40000ab200, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4002477280) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1267 +0xf8 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 171 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x40000b16b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:224 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 172 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x40000b16b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:612 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 173 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x40000b16b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:662 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 174 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x40000b16b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:752 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 175 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).processSchedIndexes(0x40000b16b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:296 +0x44 created by github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:218 +0x3c goroutine 176 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004eaa700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:249 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 214 [select]: net/http.(*persistConn).roundTrip(0x4004e4e120, 0x4000081c00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004e46600, 0xf8, 0x10a8660, 0x4004cf6588) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004e46600, 0x1b6e900, 0xc03eb91f7155f8ca, 0x1bfa82f0df) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004e46500, 0x12db048, 0x1b6e900, 0xc03eb91f7155f8ca, 0x1bfa82f0df, 0x1be4a80, 0x0, 0x40003f22f8, 0x4004e10bd0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004e10ba0, 0x4004e46500, 0xc03eb91f7155f8ca, 0x1bfa82f0df, 0x1be4a80, 0x40003f22f8, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004e10ba0, 0x4004e46500, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004e48360, 0x1b, 0x4004cf6cf0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x400239b680, 0x112f905, 0x6, 0x12db908, 0x40001491a0, 0xee40c0, 0x400009ee98, 0x4004cf6cf0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x40023dc100, 0x46, 0x12db908, 0x40001491a0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x40023dc100, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x4004e10ae0, 0xffff85e36658) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004e694b0, 0x1063520, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x400038afc0, 0x400038afc0, 0x40023dc100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.FetchNewClusterInfoCache(0xffffda80d94e, 0xe, 0x113075a, 0x7, 0x1130e44, 0x8, 0x4004e550c4, 0x2, 0xf01) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:123 +0x134 github.com/couchbase/indexing/secondary/manager.(*IndexManager).FetchNewClusterInfoCache(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:306 github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).getIndexMetadata(0x1be5520, 0x12ec7a0, 0x4000081bc0, 0x4000358a00, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:1284 +0x74 github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).handleIndexMetadataRequest(0x1be5520, 0x12edbe0, 0x40001881c0, 0x4004e46400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:1271 +0x3e8 net/http.HandlerFunc.ServeHTTP(0x4004730c60, 0x12edbe0, 0x40001881c0, 0x4004e46400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2069 +0x40 net/http.(*ServeMux).ServeHTTP(0x4004734440, 0x12edbe0, 0x40001881c0, 0x4004e46400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2448 +0x190 net/http.serverHandler.ServeHTTP(0x400447cc40, 0x12edbe0, 0x40001881c0, 0x4004e46400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2887 +0xbc net/http.(*conn).serve(0x40001bda40, 0x12f86a0, 0x4000081a40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:1952 +0x71c created by net/http.(*Server).Serve /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:3013 +0x318 goroutine 178 [select]: net/http.(*persistConn).roundTrip(0x40024bb8c0, 0x4002477740, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004eaa900, 0xf8, 0x10a8660, 0x40044e8dd8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004eaa900, 0x1b6e900, 0xc03eb91f714e585e, 0x1bfa7b506a) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004eaa800, 0x12db048, 0x1b6e900, 0xc03eb91f714e585e, 0x1bfa7b506a, 0x1be4a80, 0x0, 0x400013e9d8, 0x4004ec8060, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004ec8030, 0x4004eaa800, 0xc03eb91f714e585e, 0x1bfa7b506a, 0x1be4a80, 0x400013e9d8, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004ec8030, 0x4004eaa800, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4000043e30, 0x1b, 0x40044e9540, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004bbb680, 0x112f905, 0x6, 0x12db908, 0x4000047080, 0xee40c0, 0x4004eae498, 0x40044e9540, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x4004bc0400, 0x46, 0x12db908, 0x4000047080, 0x40013eb598, 0x81ee08, 0x40013eb748, 0x95dcbc, 0x400003e180, 0x46, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x4004bc0400, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x240, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40044e9d00, 0x4004bb9f50, 0xffff85fc00e8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x40000ab680, 0xd47600, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x40000ab680, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x40000c0380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9702 +0x344 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 179 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x40000c0380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1090 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 180 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x40000c0380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1017 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 213 [select]: net/http.(*persistConn).writeLoop(0x40024bbb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 210 [runnable]: syscall.Syscall(0x3f, 0x1c, 0x4004e51000, 0x1000, 0x502, 0x4004e51000, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x1c, 0x4004e51000, 0x1000, 0x1000, 0x0, 0x12dbfe8, 0x1b7f708) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4004eae580, 0x4004e51000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 net.(*netFD).Read(0x4004eae580, 0x4004e51000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40003f22d8, 0x4004e51000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40024bb8c0, 0x4004e51000, 0x1000, 0x1000, 0x60, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000083bc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000083bc0, 0x1, 0x40045f7aa0, 0x43d584, 0x0, 0x0, 0x400477e568) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40024bb8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 183 [IO wait]: internal/poll.runtime_pollWait(0xffff4dfdeec8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004eae798, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004eae780, 0x4004e50000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004eae780, 0x4004e50000, 0x1000, 0x1000, 0x0, 0x4004e27000, 0x4004edfa38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40003f22d0, 0x4004e50000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).fill(0x40000838c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000838c0, 0x40a, 0x4004e50000, 0x0, 0xffff00001001, 0x400, 0x400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).ReadLine(0x40000838c0, 0x1000, 0xffff5c0673f8, 0x1, 0x4004edfbc8, 0x412ff8, 0x400239b440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:389 +0x30 net/textproto.(*Reader).readLineSlice(0x4004edfcf0, 0x400239b440, 0x4004edfc48, 0xffff5c0673f8, 0x2c8, 0xffff85fb5f40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/textproto/reader.go:57 +0xac net/textproto.(*Reader).ReadLine(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/textproto/reader.go:38 net/http.ReadResponse(0x40000838c0, 0x4004e46300, 0x1000, 0x4004e50000, 0x40003f22d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/response.go:161 +0x8c github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4004ec8360, 0x4004edff10, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:132 +0x2ac github.com/couchbase/cbauth/revrpc.BabysitService(0x4004edff10, 0x4004ec8360, 0x0, 0x0, 0x0, 0xd8d03c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth/service.RegisterManager(0x13007d8, 0x4004eaa700, 0x0, 0x0, 0x4004572f80, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:146 +0xa0 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).registerWithServer(0x4004eaa700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:301 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:205 +0x78 goroutine 184 [select]: net/http.(*persistConn).roundTrip(0x40024bbb00, 0x4002477780, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004eaab00, 0xf8, 0x10a8660, 0x4004572798) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004eaab00, 0x1b6e900, 0xc03eba2d71501b1a, 0x1176f81432f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004eaaa00, 0x12db048, 0x1b6e900, 0xc03eba2d71501b1a, 0x1176f81432f, 0x1be4a80, 0x0, 0x400013e9f0, 0x4004ec81e0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004ec81b0, 0x4004eaaa00, 0xc03eba2d71501b1a, 0x1176f81432f, 0x1be4a80, 0x400013e9f0, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004ec81b0, 0x4004eaaa00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004ea7520, 0x1f, 0x4004572d10, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/indexer.getWithAuth(0x4004ea7520, 0x1f, 0xe, 0x113846e, 0x11) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:3077 +0x68 github.com/couchbase/indexing/secondary/indexer.getGlobalTopology(0x4004eba060, 0xe, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:2266 +0x78 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).updateNodeList(0x4004eaa700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:220 +0x3c created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:206 +0x98 goroutine 212 [IO wait]: internal/poll.runtime_pollWait(0xffff4dfdefb0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004eae698, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004eae680, 0x4004e53000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004eae680, 0x4004e53000, 0x1000, 0x1000, 0x40003ca3e8, 0x7ff1a8, 0x40003ca4b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40003f22e0, 0x4004e53000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40024bbb00, 0x4004e53000, 0x1000, 0x1000, 0x60, 0x0, 0x4004ea7501) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000083c80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000083c80, 0x1, 0x40045f7bc0, 0x43d584, 0x0, 0xd8c91c, 0x40003ca568) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40024bbb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 218 [IO wait]: internal/poll.runtime_pollWait(0xffff4dfdecf8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400009ef98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400009ef80, 0x4004e5f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400009ef80, 0x4004e5f000, 0x1000, 0x1000, 0x40003ca3e8, 0xffff4dfe37b0, 0x88) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40003f2308, 0x4004e5f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004e4e120, 0x4004e5f000, 0x1000, 0x1000, 0x60, 0x0, 0x4004ea7501) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004e6a1e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004e6a1e0, 0x1, 0x4004e6e000, 0x43d584, 0x0, 0xd8c91c, 0x40003ca568) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004e4e120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 219 [select]: net/http.(*persistConn).writeLoop(0x4004e4e120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc Initializing write barrier = 8000 2021-08-16T13:47:54.987+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:47:54.988+00:00 [Info] Setting ipv6=false 2021-08-16T13:47:55.003+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:47:55.009+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:47:55.009+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:47:55.010+00:00 [Info] Setting log level to Info 2021-08-16T13:47:55.010+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:47:55.014+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:47:55 audit: created new audit service 2021-08-16T13:47:55.020+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:47:55.020+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:47:55.020+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:47:55.020+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:47:55.020+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:47:55.032+00:00 [Info] security context initialized 2021-08-16T13:47:55.032+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:47:55.032+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:47:55.032+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:47:55.032+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:47:55.032+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:47:55.046+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:47:55.047+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:47:55.053+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:47:55.068+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:47:55.068+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:47:55.072+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:47:55.078+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:47:55.078+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:47:55.082+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:47:55.088+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:47:55.088+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:47:55.134+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:47:55.134+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 50 us 2021-08-16T13:47:55.135+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:47:55.136+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:47:55.137+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:47:55.137+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:47:55.137+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:47:55.137+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:47:55.137+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:47:55.137+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:47:55.139+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:55.139+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:55.144+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:47:55.144+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:47:55.144+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:47:55.144+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:55.145+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:47:55.145+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:47:55.145+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:47:55.145+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:47:55.145+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:47:55.145+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:47:55.145+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:47:55.150+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 4.676905ms 2021-08-16T13:47:55.150+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:47:55.152+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:47:55.152+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:47:55.152+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:47:55.152+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:47:55.152+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:47:55.152+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:47:55.163+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:47:55.163+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:47:55.163+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:47:55.163+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:47:55.163+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x400049321f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x400049321f pc=0xbd8ea4] goroutine 173 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004d829f0 sp=0x4004d829c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004d82a30 sp=0x4004d829f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x400458d6c0, 0x4004d5a000, 0x11a9890, 0x4004d82b40, 0x400458d6d8, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004d82ab0 sp=0x4004d82a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x400458d6c0, 0x4004d5a000, 0x11a9890, 0x4004d82b40, 0x400458d6d8, 0xffff87241a98, 0x4004d82c18, 0x49) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004d82af0 sp=0x4004d82ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4004466640, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004d82da0 sp=0x4004d82af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4004466640, 0x40002410a0, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004d82e60 sp=0x4004d82da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004d28620, 0x4004d28630, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004d83200 sp=0x4004d82e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004d28620, 0x4004d28630, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004d838f0 sp=0x4004d83200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x400465bed0, 0x4004d1bc00, 0x400025d600, 0x4004d28600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004d83fb0 sp=0x4004d838f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004d83fb0 sp=0x4004d83fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x4000182000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x4000203620, 0x4, 0x1136a87, 0xf, 0x400020be68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 34 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400020c0f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 35 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400020c108) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 36 [IO wait]: internal/poll.runtime_pollWait(0xffff8720e1f8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400024c718, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400024c700, 0x4000385000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400024c700, 0x4000385000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000210098, 0x4000385000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x4000200780, 0x400048a001, 0xdff, 0xdff, 0x40001238e8, 0x6902d8, 0x4000207948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400020c150, 0x400048a001, 0xdff, 0xdff, 0x40002079f8, 0x6f1968, 0x4000207938) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x400021e840, 0xa, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400021e840, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400021e840, 0xf5a960, 0x40003feec0, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x40003feea0, 0x400022e600, 0x4000207b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x4000216870, 0x12f8f98, 0x40003feea0, 0x4000213327, 0x16, 0x4000442900, 0x4000123c01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x4000216870, 0x12f8f98, 0x40003feea0, 0x4000216870, 0x4000212b30, 0x4000212b40, 0x400024cd00, 0x400022e520, 0xee0980, 0x4000442900, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x4000216870, 0x12f8f98, 0x40003feea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4000202420, 0x4000123f78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400011ff78, 0x4000202420, 0x12db8a8, 0x400020c120, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x4000202420, 0x40002225a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x4000202420, 0x40002225a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 37 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 38 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 39 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 40 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 41 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 5 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400000a600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 42 [syscall]: syscall.Syscall(0x3f, 0x0, 0x4000249fa0, 0x4, 0x4000249e68, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x4000249fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000140000, 0x4000249fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x400013e000, 0x4000249fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 45 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400020e7a0, 0x12c18bc, 0x1, 0x40014bfe58, 0x4000038720, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x40000596c8, 0x4000038720, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40014bffa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4000080000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 25 [select]: net/http.(*persistConn).writeLoop(0x400014e6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 22 [select]: net/http.(*persistConn).readLoop(0x400000c240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 23 [select]: net/http.(*persistConn).writeLoop(0x400000c240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 46 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4000080000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 26 [IO wait]: internal/poll.runtime_pollWait(0xffff8720e110, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400009a198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400009a180, 0x400048b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400009a180, 0x400048b000, 0x1000, 0x1000, 0x4000069a88, 0x68f7a4, 0x400021ef70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010050, 0x400048b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c240, 0x400048b000, 0x1000, 0x1000, 0xf12180, 0x400008a360, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000094960) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000094960, 0xa, 0x1008880, 0x400008a320, 0x199, 0xc64, 0x4000069b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000094960, 0x2, 0x0, 0x9, 0x56b, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000096ff0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000096ff0, 0x40000a0000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40000802c0, 0x40000a0000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40000802c0, 0x40000a0000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000080300, 0x40000a0000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x400021ef20, 0x0, 0x4000069eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400021ef20, 0x0, 0x0, 0x4000069f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400021ef20, 0xee0d40, 0x400008a320, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400023def0, 0x4000094de0, 0x4000038d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 24 [IO wait]: internal/poll.runtime_pollWait(0xffff8720e028, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400024c098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400024c080, 0x40001c0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400024c080, 0x40001c0000, 0x1000, 0x1000, 0x40014b2be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010058, 0x40001c0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400014e6c0, 0x40001c0000, 0x1000, 0x1000, 0x40014b2cf8, 0x67867c, 0x400010e360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000094ae0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000094ae0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400014e6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 31 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000258fc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 12 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400020e7a0, 0x1144559, 0x1b, 0x4004c4de38, 0x40046157a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x400006d7f0, 0x40046157a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004c4df80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40044638c0, 0x400006d7f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 11 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x40014febe0, 0x4004615680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 8 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000258ec0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 10 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000259b80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 146 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40000b4620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 67 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x4000188620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 105 [IO wait]: internal/poll.runtime_pollWait(0xffff8720d630, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400024c298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400024c280, 0x40001f1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400024c280, 0x40001f1000, 0x1000, 0x1000, 0x4000123688, 0x69f178, 0x40001236f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000210158, 0x40001f1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400027d0e0, 0x40001f1000, 0x1000, 0x1000, 0x5, 0x40001a0000, 0x40001237c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004580840) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004580840, 0x68f90a, 0x4000123808, 0x4344e0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004580840, 0x41c330, 0xffff872cb6c0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40000aacf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40000aacf0, 0x400007f000, 0x1000, 0x1000, 0x4000123998, 0x44e884, 0x1c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000080640, 0x400007f000, 0x1000, 0x1000, 0x40000945f8, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000080640, 0x400007f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000080680, 0x400007f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000123d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000123d80, 0xa, 0x400007f201, 0x1, 0xffff87212880, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000123d80, 0x400007f20a, 0x1, 0x40014c9690, 0x0, 0x400007f259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000123d80, 0x40014c260a, 0x40014c9690, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4004410380, 0x400442d360, 0x1f, 0x4000123ee0, 0x4004425d58, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x4004410380, 0x113075a, 0x7, 0x400442e5f0, 0xd, 0x4004425d58, 0x0, 0x0, 0x2) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x4004410340, 0x400442e5f0, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 64 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40000b4620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 70 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x4000188620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 71 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x4000188620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 72 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x4000188620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 93 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400455b000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 32 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x40001416e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 33 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x400044ff80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 82 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x400009b000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 83 [IO wait]: internal/poll.runtime_pollWait(0xffff8720dab8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400009b118, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400009b100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400009b100, 0x0, 0x4000243698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4000131518, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4000131518, 0x12ed9d0, 0x4000131518, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x400009b080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 84 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4000215e10, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 85 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4000215e10, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 86 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x4000215e10) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 87 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x4004504a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 79 [select]: net/http.(*persistConn).readLoop(0x400027c900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 80 [select]: net/http.(*persistConn).writeLoop(0x400027c900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 88 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x4000180380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 99 [select]: net/http.(*persistConn).readLoop(0x400027cb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 100 [select]: net/http.(*persistConn).writeLoop(0x400027cb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 51 [IO wait]: internal/poll.runtime_pollWait(0xffff8720d718, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400024c118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400024c100, 0x4004501000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400024c100, 0x4004501000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000210090, 0x4004501000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400027cd80, 0x4004501000, 0x1000, 0x1000, 0x4000102f00, 0x200000003, 0x4000102f00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004580300) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004580300, 0x40000a600a, 0x40001258c8, 0x4d3b10, 0x4000140060, 0x40000a6000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004580300, 0x0, 0x1, 0x0, 0x4000140078, 0x40000a6000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40000aa720) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40000aa720, 0x400048c000, 0x1000, 0x1000, 0x40001259b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000080400, 0x400048c000, 0x1000, 0x1000, 0x40003ff5b8, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000080400, 0x400048c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000080440, 0x400048c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000125dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000125dd0, 0xa, 0x400048c201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000125dd0, 0x400048c20a, 0x1, 0x400442e5d9, 0x0, 0x400048c2e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000125dd0, 0x400049000a, 0x400442e5d9, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4004410380, 0x400003a3d8, 0x17, 0x4000125f20, 0x40044cec78, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x4004410380, 0x113075a, 0x7, 0x40044cec78, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x4004410340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 52 [IO wait]: internal/poll.runtime_pollWait(0xffff8720d800, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400024d298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400024d280, 0x40044d4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400024d280, 0x40044d4000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000210598, 0x40044d4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400027c900, 0x40044d4000, 0x1000, 0x1000, 0x4000103080, 0x200000003, 0x4000103080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4001497440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4001497440, 0x40000a600a, 0x40014c18c8, 0x4d3b10, 0x4000140060, 0x40000a6000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4001497440, 0x0, 0x1, 0x0, 0x4000140078, 0x40000a6000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40000aa450) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40000aa450, 0x40000a1000, 0x1000, 0x1000, 0x40014c19b8, 0x0, 0x40014c1988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000080340, 0x40000a1000, 0x1000, 0x1000, 0x4000102c00, 0x3, 0x40014c1be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000080340, 0x40000a1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000080380, 0x40000a1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40014c1dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40014c1dd0, 0xa, 0x40000a1201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40014c1dd0, 0x40000a120a, 0x1, 0x400442e37b, 0x0, 0x40000a1263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40014c1dd0, 0x400022e00a, 0x400442e37b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4004410380, 0x40000421b0, 0x24, 0x40014c1f20, 0x40044cec60, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x4004410380, 0x113075a, 0x7, 0x40044cec60, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x4004410340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 61 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40000801c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 55 [select]: net/http.(*persistConn).readLoop(0x400027cd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 56 [select]: net/http.(*persistConn).writeLoop(0x400027cd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 59 [select]: net/http.(*persistConn).readLoop(0x400027d0e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 60 [select]: net/http.(*persistConn).writeLoop(0x400027d0e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 63 [semacquire]: sync.runtime_Semacquire(0x400465bed8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x400465bed0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x400025d600, 0x400011b260, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xfffff4387969, 0x29, 0xfffff43879cb, 0x25, 0x4004615ec0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004be6c30, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xfffff4387969, 0x29, 0xfffff43879cb, 0x25, 0x4004615ec0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004be6c30, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004d83190, 0x4004d82e60, 0x400011b260, 0x4000493b00, 0x4000258ec0, 0x0, 0x1bf, 0x4004d82da8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x4000182000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004be6c30, 0xb, 0x4004be6c28, 0x6, 0x4004be6c40, 0xd, 0x4004bbee60, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x4000182000, 0x40003f0000, 0x4000096270) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x4000182000, 0x4000483260, 0x2, 0x2, 0x4000483270, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 94 [IO wait]: internal/poll.runtime_pollWait(0xffff8720d9d0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40014f8f18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40014f8f00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40014f8f00, 0x12d9d48, 0x40014d8640, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004bc0e10, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004bc0e10, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x40014d9580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 95 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x40014d99c0, 0x40014dad20, 0x40014dad90, 0x4004bcc890, 0x12e8f90, 0x40044647e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x400473bf44, 0x5, 0x40014d9580, 0x12eca70, 0x40014d9440, 0x130d478, 0x40002200e0, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x4000188e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x4000188e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 96 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40044647e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 97 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40044647e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 114 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x40000d8090, 0x4004728f00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 115 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x4000080140) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 116 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x40014febe0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 117 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x40014febe0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 118 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x40014e83d8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 119 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x40014fec80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 120 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x40014fec80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 121 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x40014d99c0, 0x4004bcc890) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 13 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 14 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400014aa00, 0xd00007, 0x400465ac50, 0x5, 0x400006d800, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 15 [IO wait]: internal/poll.runtime_pollWait(0xffff8720d460, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400009b418, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400009b400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400009b400, 0xffff872c0108, 0x30, 0x4004d1c2d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40046511b8, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40046511b8, 0x4004d1c2d0, 0x11ae500, 0x4004464940, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40044648c0, 0x12ed9d0, 0x40046511b8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40044648c0, 0x12ed9d0, 0x40046511b8, 0x4000182000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 16 [IO wait]: internal/poll.runtime_pollWait(0xffff8720d378, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400009b518, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400009b500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400009b500, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40046511d0, 0xffff872c0108, 0xffff472bd7e8, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40046511d0, 0x665e1c, 0x0, 0xfe5e60, 0x4004664dc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x40046511e8, 0x4004d1c360, 0x11ae500, 0x4004464a20, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40044649a0, 0x12ec710, 0x40046511e8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40044649a0, 0x12ec710, 0x40046511e8, 0x4000182000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 62 [IO wait]: internal/poll.runtime_pollWait(0xffff8720d8e8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400024d418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400024d400, 0x40044f2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400024d400, 0x40044f2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40002105c0, 0x40044f2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400027cb40, 0x40044f2000, 0x1000, 0x1000, 0x20, 0x30, 0x20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40014978c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40014978c0, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40014978c0, 0x7356e4, 0xfffff438794e, 0xe, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40044f1da0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40044f1da0, 0x40001c5800, 0x200, 0x200, 0xfffff438794e, 0xe, 0x4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000259f00, 0x40001c5800, 0x200, 0x200, 0x40001c5800, 0x200, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000259f00, 0x40001c5800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000259f40, 0x40001c5800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40045a6f20, 0x0, 0x40014a36b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40045a6f20, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40045a6f20, 0xee0d40, 0x400008ac30, 0x12db068, 0x4000259f40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40045a8480, 0x4004581020, 0x40045ba240) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 123 [IO wait]: internal/poll.runtime_pollWait(0xffff475f8168, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40014f9298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40014f9280, 0x4004bff000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40014f9280, 0x4004bff000, 0x1000, 0x1000, 0x4004d14be0, 0x469b10, 0x400149a480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40014e8430, 0x4004bff000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000443680, 0x4004bff000, 0x1000, 0x1000, 0x4004d14cf8, 0x67867c, 0x40047294a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400473cd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x400473cd80, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4000443680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 124 [select]: net/http.(*persistConn).writeLoop(0x4000443680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 65 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40000b4620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 130 [IO wait]: internal/poll.runtime_pollWait(0xffff475f8080, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000b7118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000b7100, 0x4004d91000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000b7100, 0x4004d91000, 0x1000, 0x1000, 0x40014b5be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000102b8, 0x4004d91000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40004437a0, 0x4004d91000, 0x1000, 0x1000, 0x40014b5cf8, 0x67867c, 0x4004dc6000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004b7f0e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004b7f0e0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40004437a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 131 [select]: net/http.(*persistConn).writeLoop(0x40004437a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 147 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40000b4620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 148 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4000161980) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 149 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40045de000, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 150 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40045de000, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 151 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x40045de000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 152 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40000b4620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 153 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400020e7a0, 0x1144559, 0x1b, 0x4004597e38, 0x40045baa20, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x400020fe70, 0x40045baa20, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004597f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004581500, 0x400020fe70) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 158 [IO wait]: internal/poll.runtime_pollWait(0xffff8720d548, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40045f2098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40045f2080, 0x40045b5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40045f2080, 0x40045b5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000210670, 0x40045b5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400027de60, 0x40045b5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004581d40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004581d40, 0xa, 0x0, 0x0, 0x0, 0x0, 0x4000054438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004581d40, 0x0, 0x4000054478, 0x655784, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40045c4e40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40045c4e40, 0x40045bea00, 0x200, 0x200, 0x4, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000161b80, 0x40045bea00, 0x200, 0x200, 0x40044d1987, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000161b80, 0x40045bea00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000161bc0, 0x40045bea00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40045dcc60, 0x0, 0x40000546b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40045dcc60, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40045dcc60, 0xee0d40, 0x400008b090, 0x12db068, 0x4000161bc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40045a9050, 0x40045fa120, 0x40045bb800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 156 [select]: net/http.(*persistConn).readLoop(0x400027de60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 157 [select]: net/http.(*persistConn).writeLoop(0x400027de60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 162 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001619c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 163 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x40014c5e40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 164 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x40014c5e40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 165 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x40014c5e40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 166 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x40014c5e40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 167 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004d34200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 168 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x4004d34200, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 169 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x4000182000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 170 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x4000182000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 171 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x4000182000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 172 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:48:00.323+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:48:00.323+00:00 [Info] Setting ipv6=false 2021-08-16T13:48:00.328+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:48:00.332+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:48:00.332+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:48:00.332+00:00 [Info] Setting log level to Info 2021-08-16T13:48:00.332+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:48:00.335+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021-08-16T13:48:00.343+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:48:00.343+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:48:00.343+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:48:00.343+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:48:00.343+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021/08/16 13:48:00 audit: created new audit service 2021-08-16T13:48:00.355+00:00 [Info] security context initialized 2021-08-16T13:48:00.355+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:48:00.355+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:48:00.355+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:48:00.355+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:48:00.355+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:48:00.368+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:48:00.368+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:48:00.381+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:48:00.394+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:48:00.394+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:48:00.396+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:48:00.397+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:48:00.401+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:48:00.406+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:48:00.408+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:48:00.408+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:48:00.448+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:48:00.449+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 55 us 2021-08-16T13:48:00.449+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:48:00.451+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:48:00.451+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:48:00.452+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:48:00.452+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:48:00.452+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:48:00.452+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:48:00.452+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:48:00.458+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:00.458+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:00.459+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:48:00.459+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:48:00.459+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:48:00.459+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:00.459+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:48:00.459+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:48:00.459+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:48:00.459+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:48:00.459+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:48:00.459+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:48:00.459+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:48:00.464+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 4.899639ms 2021-08-16T13:48:00.465+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:48:00.466+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:48:00.466+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:48:00.466+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:00.466+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:48:00.466+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:48:00.466+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:48:00.477+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:48:00.477+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:48:00.477+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:48:00.477+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:48:00.478+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x4004cd0fdf fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x4004cd0fdf pc=0xbd8ea4] goroutine 170 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004c729f0 sp=0x4004c729c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004c72a30 sp=0x4004c729f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x4000405a00, 0x40045fd200, 0x11a9890, 0x4004c72b40, 0x4000405a18, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004c72ab0 sp=0x4004c72a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x4000405a00, 0x40045fd200, 0x11a9890, 0x4004c72b40, 0x4000405a18, 0xc18f8c, 0x4004c72c18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004c72af0 sp=0x4004c72ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4000224b40, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004c72da0 sp=0x4004c72af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4000224b40, 0x400440d6c0, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004c72e60 sp=0x4004c72da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004cb5b40, 0x4004cb5b50, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004c73200 sp=0x4004c72e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004cb5b40, 0x4004cb5b50, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004c738f0 sp=0x4004c73200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004dabe30, 0x4004cff880, 0x4000250680, 0x4004cb5b20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004c73fb0 sp=0x4004c738f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004c73fb0 sp=0x4004c73fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x40001c0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x40002054a0, 0x4, 0x1136a87, 0xf, 0x400019de68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 18 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400020e108) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 19 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400020e120) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 20 [IO wait]: internal/poll.runtime_pollWait(0xffffb1a96e88, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011e098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011e080, 0x4000139000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011e080, 0x4000139000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000110008, 0x4000139000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x40001120c0, 0x4000138001, 0xdff, 0xdff, 0x40000918e8, 0x6902d8, 0x4000153948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x4000118030, 0x4000138001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x4000136160, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4000136160, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4000136160, 0xf5a960, 0x4000082860, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4000082840, 0x4000046240, 0x4000153b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x40000ac000, 0x12f8f98, 0x4000082840, 0x4000036b67, 0x16, 0x400000c5a0, 0x4000091c01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x40000ac000, 0x12f8f98, 0x4000082840, 0x40000ac000, 0x40000361a0, 0x40000361d0, 0x400011e000, 0x4000046020, 0xee0980, 0x400000c5a0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x40000ac000, 0x12f8f98, 0x4000082840) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4000204420, 0x4000091f78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400019bf78, 0x4000204420, 0x12db8a8, 0x4000118000, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x4000204420, 0x40002223c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x4000204420, 0x40002223c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 21 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 22 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 23 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 24 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 25 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 7 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400000a700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 26 [syscall]: syscall.Syscall(0x3f, 0x0, 0x4000055fa0, 0x4, 0x4000055e68, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x4000055fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x4000055fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x4000055fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 40 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001ab340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 56 [IO wait]: internal/poll.runtime_pollWait(0xffffb1a96da0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000212198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000212180, 0x40001fa000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000212180, 0x40001fa000, 0x1000, 0x1000, 0x4000019a88, 0x68f7a4, 0x400021e730) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001a2048, 0x40001fa000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400026c480, 0x40001fa000, 0x1000, 0x1000, 0xf12180, 0x4000180130, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000192540) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000192540, 0xa, 0x1008880, 0x40001800f0, 0x199, 0x1000, 0x4000019b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000192540, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000194b70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000194b70, 0x40000e8000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40001aa100, 0x40000e8000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40001aa100, 0x40000e8000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001aa140, 0x40000e8000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x400021e6e0, 0x0, 0x4000019eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400021e6e0, 0x0, 0x0, 0x4000019f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400021e6e0, 0xee0d40, 0x40001800f0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40001261b0, 0x40001926c0, 0x4000038420) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 54 [select]: net/http.(*persistConn).readLoop(0x400026c480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 55 [select]: net/http.(*persistConn).writeLoop(0x400026c480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 9 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40002066e0, 0x12c18bc, 0x1, 0x400019de58, 0x4000182720, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x40000f6ec8, 0x4000182720, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400019dfa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x40001aa0c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 10 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x40001aa0c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 109 [IO wait]: internal/poll.runtime_pollWait(0xffffb1a960f0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011f018, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011f000, 0x400251f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011f000, 0x400251f000, 0x1000, 0x1000, 0x400256ea48, 0x652ce0, 0x12edac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000106e8, 0x400251f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000d320, 0x400251f000, 0x1000, 0x1000, 0x113674e, 0x412bc0, 0x40000ac6e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400252a2a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400252a2a0, 0x400023190a, 0x2, 0x4, 0x4000231920, 0x2, 0x4002370438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400252a2a0, 0x7356e4, 0xffffd9e6f94e, 0xe, 0x4, 0x40002319c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40046cb1d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40046cb1d0, 0x4004c33200, 0x200, 0x200, 0xffffd9e6f94e, 0xe, 0x4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400247f580, 0x4004c33200, 0x200, 0x200, 0x4000043537, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400247f580, 0x4004c33200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400247f5c0, 0x4004c33200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004c358c0, 0x4000231d00, 0x40023706b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004c358c0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004c358c0, 0xee0d40, 0x40000acff0, 0x12db068, 0x400247f5c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40046cd5f0, 0x400252ba40, 0x4004c04b40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 13 [IO wait]: internal/poll.runtime_pollWait(0xffffb1a96cb8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000c8098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000c8080, 0x40000ca000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000c8080, 0x40000ca000, 0x1000, 0x1000, 0x4000013be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000110060, 0x40000ca000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400001c360, 0x40000ca000, 0x1000, 0x1000, 0x4000013cf8, 0x67867c, 0x4000182c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001126c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40001126c0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400001c360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 14 [select]: net/http.(*persistConn).writeLoop(0x400001c360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 61 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001aaf40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 128 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x40046542c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 108 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400247e400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 98 [IO wait]: internal/poll.runtime_pollWait(0xffffb1a96660, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000212d18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000212d00, 0x400445e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000212d00, 0x400445e000, 0x1000, 0x1000, 0x4000467688, 0x69f178, 0x40004676f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001a2098, 0x400445e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400441c360, 0x400445e000, 0x1000, 0x1000, 0xffff885bafff, 0x4002402500, 0x40004677e8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003fbce0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003fbce0, 0xa, 0xffff883e3c00, 0xffff885bafff, 0x4000467838, 0x41cff4, 0xffff883e3b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003fbce0, 0x41c330, 0xffffb1aa2ea0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40024a8e40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40024a8e40, 0x4004468000, 0x1000, 0x1000, 0x4000467998, 0x44e884, 0x4000467948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40001aa800, 0x4004468000, 0x1000, 0x1000, 0x4000082e98, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40001aa800, 0x4004468000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001aa880, 0x4004468000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000467d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000467d80, 0xa, 0x4004468201, 0x1, 0xda7, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000467d80, 0x400446820a, 0x1, 0x40023f3d9a, 0x0, 0x4004468259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000467d80, 0x400021f00a, 0x40023f3d9a, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000dae10, 0x40044f17c0, 0x1f, 0x4000467ee0, 0x40044f3320, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x40000dae10, 0x113075a, 0x7, 0x4002565b90, 0xd, 0x40044f3320, 0x0, 0x4714f4, 0x40000dae10) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x40000dadd0, 0x4002565b90, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 63 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x400043c8c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 31 [select]: net/http.(*persistConn).readLoop(0x400441c360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 32 [select]: net/http.(*persistConn).writeLoop(0x400441c360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 155 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40046c91e0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 66 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x400043c8c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 67 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x400043c8c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 68 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x400043c8c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 33 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40002066e0, 0x1144559, 0x1b, 0x4000175e38, 0x4004610420, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x40024a7a50, 0x4004610420, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x40024a8b70) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000175f80, 0x0, 0x1000000010000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004516900, 0x40024a7a50) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 103 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001aa580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 41 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x4000112ba0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 42 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x400043e9f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 43 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x40000c8000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 44 [IO wait]: internal/poll.runtime_pollWait(0xffffb1a96578, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000c8718, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000c8700, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000c8700, 0x0, 0x4000059698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40044a3a70, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40044a3a70, 0x12ed9d0, 0x40044a3a70, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x40000c8180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 45 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400013d450, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 46 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400013d450, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 47 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x400013d450) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 48 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x400043fe60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 87 [select]: net/http.(*persistConn).writeLoop(0x40004590e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 16 [IO wait]: internal/poll.runtime_pollWait(0xffffb1a96490, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004450198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004450180, 0x40000e9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004450180, 0x40000e9000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001a2088, 0x40000e9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000458900, 0x40000e9000, 0x1000, 0x1000, 0x4000241e00, 0x200000003, 0x4000241e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003fbc20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003fbc20, 0x400040000a, 0x40000918c8, 0x4d3b10, 0x40000820c0, 0x4000400000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003fbc20, 0x0, 0x1, 0x0, 0x40000820d8, 0x4000400000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000478a80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000478a80, 0x400257c000, 0x1000, 0x1000, 0x40000919b8, 0x0, 0x4000091988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000210500, 0x400257c000, 0x1000, 0x1000, 0x4000186f00, 0x3, 0x4000091be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000210500, 0x400257c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000210580, 0x400257c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000091dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000091dd0, 0xa, 0x400257c201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000091dd0, 0x400257c20a, 0x1, 0x40023f388b, 0x0, 0x400257c263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000091dd0, 0x400020020a, 0x40023f388b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000dae10, 0x4000042ae0, 0x24, 0x4000091f20, 0x400255ca20, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x40000dae10, 0x113075a, 0x7, 0x400255ca20, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x40000dadd0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 83 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x4000276690) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 15 [IO wait]: internal/poll.runtime_pollWait(0xffffb1a963a8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004450298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004450280, 0x40024fe000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004450280, 0x40024fe000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000208158, 0x40024fe000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000458d80, 0x40024fe000, 0x1000, 0x1000, 0x4000102f00, 0x200000003, 0x4000102f00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000192d20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000192d20, 0x400040000a, 0x40001978c8, 0x4d3b10, 0x40000820c0, 0x4000400000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000192d20, 0x0, 0x1, 0x0, 0x40000820d8, 0x4000400000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000478cf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000478cf0, 0x4002377000, 0x1000, 0x1000, 0x40001979b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000210640, 0x4002377000, 0x1000, 0x1000, 0x4000082e98, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000210640, 0x4002377000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000210680, 0x4002377000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000197dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000197dd0, 0xa, 0x4002377201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000197dd0, 0x400237720a, 0x1, 0x4002565b79, 0x0, 0x40023772e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000197dd0, 0x40000db60a, 0x4002565b79, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000dae10, 0x40044084b0, 0x17, 0x4000197f20, 0x400019e660, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x40000dae10, 0x113075a, 0x7, 0x400019e660, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x40000dadd0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 29 [select]: net/http.(*persistConn).readLoop(0x4000458900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 30 [select]: net/http.(*persistConn).writeLoop(0x4000458900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 88 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40000e0a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 75 [select]: net/http.(*persistConn).readLoop(0x4000458d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 76 [select]: net/http.(*persistConn).writeLoop(0x4000458d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 86 [select]: net/http.(*persistConn).readLoop(0x40004590e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 147 [IO wait]: internal/poll.runtime_pollWait(0xffff8833b968, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004450998, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004450980, 0x4004c76000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004450980, 0x4004c76000, 0x1000, 0x1000, 0x400256fbe0, 0x469b10, 0x4000187800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010878, 0x4004c76000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400026c900, 0x4004c76000, 0x1000, 0x1000, 0x400256fcf8, 0x67867c, 0x4004c050e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004c442a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004c442a0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400026c900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 106 [select]: net/http.(*persistConn).readLoop(0x400000d320) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 107 [select]: net/http.(*persistConn).writeLoop(0x400000d320) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 79 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400247eb00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 80 [IO wait]: internal/poll.runtime_pollWait(0xffffb1a961d8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004450698, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4004450680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4004450680, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c8b590, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c8b590, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x400440b380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 81 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x400440b7c0, 0x400440cc40, 0x400440ccb0, 0x4004cb46e0, 0x12e8f90, 0x40044c28c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x40045ef804, 0x5, 0x400440b380, 0x12eca70, 0x400440b240, 0x130d478, 0x40000d8310, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x400043c380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x400043c380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 114 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40044c28c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 115 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40044c28c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 116 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x4002456750, 0x4004587320) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 117 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x40001aa540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 118 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x4004428a00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 119 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x4004428a00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 120 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x4000208498) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 121 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x4004428aa0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 122 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x4004428aa0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 123 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x400440b7c0, 0x4004cb46e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 124 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x4004428a00, 0x4004587440) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 130 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 131 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x40001805f0, 0xd00007, 0x40024725d0, 0x5, 0x40024a7a60, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 132 [IO wait]: internal/poll.runtime_pollWait(0xffffb1a96748, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000212f98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000212f80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000212f80, 0xffffb1b12a68, 0x30, 0x4004cf01b0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004616210, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004616210, 0x4004cf01b0, 0x11ae500, 0x40000e0860, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40000e07e0, 0x12ed9d0, 0x4004616210, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40000e07e0, 0x12ed9d0, 0x4004616210, 0x40001c0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 133 [IO wait]: internal/poll.runtime_pollWait(0xffffb1a96008, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000213118, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000213100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000213100, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004616228, 0xffffb1b12a68, 0xffff88331e08, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004616228, 0x665e1c, 0x0, 0xfe5e60, 0x40000f6dc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004616240, 0x4004cf0240, 0x11ae500, 0x40000e0940, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40000e08c0, 0x12ec710, 0x4004616240, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40000e08c0, 0x12ec710, 0x4004616240, 0x40001c0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 129 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x40046542c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 135 [semacquire]: sync.runtime_Semacquire(0x4004dabe38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004dabe30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x4000250680, 0x400043e5a0, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffd9e6f969, 0x29, 0xffffd9e6f9cb, 0x25, 0x4004db26c0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004cbe4f0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffd9e6f969, 0x29, 0xffffd9e6f9cb, 0x25, 0x4004db26c0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004cbe4f0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004c73190, 0x4004c72e60, 0x400043e5a0, 0x4000227d40, 0x40001aaf40, 0x0, 0x1bf, 0x4000468da8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x40001c0000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004cbe4f0, 0xb, 0x4004cbe4e8, 0x6, 0x4004cbe500, 0xd, 0x4004c8ce00, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x40001c0000, 0x40003f0000, 0x4000194360) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x40001c0000, 0x4000116620, 0x2, 0x2, 0x4000116630, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 148 [select]: net/http.(*persistConn).writeLoop(0x400026c900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 127 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001abac0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 156 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40046c91e0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 157 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x40046c91e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 136 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40000e0a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 137 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40000e0a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 138 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40000e0a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 139 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40000e0a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 140 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x40001aba80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 89 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40002066e0, 0x1144559, 0x1b, 0x4000097e38, 0x4004611020, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x40047a26c0, 0x4004611020, 0x4000458d80, 0x40001b1640) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x12c2e00, 0x4000472e01) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000097f80, 0x12c2e00, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004517140, 0x40047a26c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 92 [IO wait]: internal/poll.runtime_pollWait(0xffff8833b880, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000c9318, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000c9300, 0x40047d3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000c9300, 0x40047d3000, 0x1000, 0x1000, 0x4002424be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000110230, 0x40047d3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400001d440, 0x40047d3000, 0x1000, 0x1000, 0x4002424cf8, 0x67867c, 0x4000039bc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004732f60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004732f60, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400001d440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 93 [select]: net/http.(*persistConn).writeLoop(0x400001d440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 141 [IO wait]: internal/poll.runtime_pollWait(0xffffb1a962c0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004450398, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004450380, 0x400257d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004450380, 0x400257d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000110148, 0x400257d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40004590e0, 0x400257d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001139e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001139e0, 0x40003e960a, 0x40003f9b00, 0x4000233ce0, 0x400027e420, 0x400027fbc0, 0x40000f8c38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40001139e0, 0x40003f9800, 0x40000f8c78, 0x655784, 0x40003fa8a0, 0x400027e120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40046509f0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40046509f0, 0x400460d600, 0x200, 0x200, 0x4, 0x0, 0x40000f8d08) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40001abb40, 0x400460d600, 0x200, 0x200, 0x40001ad9b7, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40001abb40, 0x400460d600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001abb80, 0x400460d600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004657080, 0x4000103500, 0x40000f8eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004657080, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004657080, 0xee0d40, 0x4000180910, 0x12db068, 0x40001abb80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004652750, 0x40045172c0, 0x4004611560) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 162 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x40046542c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 163 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x40046542c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 164 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004c96f00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 165 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x4004c96f00, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 166 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x40001c0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 167 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x40001c0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 168 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x40001c0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 169 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:48:05.637+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:48:05.637+00:00 [Info] Setting ipv6=false 2021-08-16T13:48:05.650+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:48:05.653+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:48:05.653+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:48:05.653+00:00 [Info] Setting log level to Info 2021-08-16T13:48:05.655+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021-08-16T13:48:05.656+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021/08/16 13:48:05 audit: created new audit service 2021-08-16T13:48:05.665+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:48:05.665+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:48:05.665+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:48:05.665+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:48:05.665+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:48:05.678+00:00 [Info] security context initialized 2021-08-16T13:48:05.678+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:48:05.678+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:48:05.678+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:48:05.678+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:48:05.678+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:48:05.691+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:48:05.693+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:48:05.693+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:48:05.719+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:48:05.720+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:48:05.721+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:48:05.721+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:48:05.723+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:48:05.724+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:48:05.725+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:48:05.725+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:48:05.772+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:48:05.772+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 51 us 2021-08-16T13:48:05.772+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:48:05.774+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:48:05.774+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:48:05.775+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:48:05.775+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:48:05.775+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:48:05.775+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:48:05.775+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:48:05.777+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:05.777+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:05.780+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:48:05.780+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:48:05.780+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:48:05.780+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:05.780+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:48:05.780+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:48:05.780+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:48:05.780+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:48:05.780+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:48:05.780+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:48:05.780+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:48:05.785+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 4.53796ms 2021-08-16T13:48:05.785+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:48:05.786+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:48:05.786+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:05.786+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:48:05.786+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:48:05.786+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:48:05.786+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... unexpected fault address 0x4004cc0d9f fatal error: fault 2021-08-16T13:48:05.797+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:48:05.797+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:48:05.798+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:48:05.798+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:48:05.798+00:00 [Info] : Recover shard shard1 from metadata completed. [signal SIGBUS: bus error code=0x1 addr=0x4004cc0d9f pc=0xbd8ea4] goroutine 204 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004d789f0 sp=0x4004d789c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004d78a30 sp=0x4004d789f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x4004ce2680, 0x400481b800, 0x11a9890, 0x4004d78b40, 0x4004ce2698, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004d78ab0 sp=0x4004d78a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x4004ce2680, 0x400481b800, 0x11a9890, 0x4004d78b40, 0x4004ce2698, 0xc18f8c, 0x4004d78c18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004d78af0 sp=0x4004d78ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x40001af040, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004d78da0 sp=0x4004d78af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x40001af040, 0x400007fa40, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004d78e60 sp=0x4004d78da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004cd2e50, 0x4004cd2e60, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004d79200 sp=0x4004d78e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004cd2e50, 0x4004cd2e60, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004d798f0 sp=0x4004d79200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004cff700, 0x4004eb6a80, 0x40000c3b80, 0x4004cd2e30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004d79fb0 sp=0x4004d798f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004d79fb0 sp=0x4004d79fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x4000154000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x40000958c0, 0x4, 0x1136a87, 0xf, 0x4000119e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 5 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400000e120) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 6 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400000e138) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 7 [IO wait]: internal/poll.runtime_pollWait(0xffffaec12e88, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400012a098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400012a080, 0x4000145000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400012a080, 0x4000145000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011e010, 0x4000145000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x400010e120, 0x4000216001, 0xdff, 0xdff, 0x400019d8e8, 0x6902d8, 0x4000115948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400011a048, 0x4000216001, 0xdff, 0xdff, 0x40001159f8, 0x6f1968, 0x4000115938) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40001e6840, 0xa, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40001e6840, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40001e6840, 0xf5a960, 0x40001ba440, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x40001ba420, 0x40001c22c0, 0x4000115b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x40001801e0, 0x12f8f98, 0x40001ba420, 0x40001b4c37, 0x16, 0x40001c7200, 0x400019dc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x40001801e0, 0x12f8f98, 0x40001ba420, 0x40001801e0, 0x40001b4448, 0x40001b4450, 0x4000208080, 0x40001c21e0, 0xee0980, 0x40001c7200, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x40001801e0, 0x12f8f98, 0x40001ba420) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4000094840, 0x400019df78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400019df78, 0x4000094840, 0x12db8a8, 0x400011a018, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x4000094840, 0x40000a4460, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x4000094840, 0x40000a4460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 8 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 9 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 10 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 11 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 12 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 34 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400010c200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 13 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40004357a0, 0x4, 0x4000435668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40004357a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x40001ba000, 0x40004357a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x40001b8000, 0x40004357a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 39 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400006c7d0, 0x12c18bc, 0x1, 0x4000243e58, 0x4000130720, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x4000433ec8, 0x4000130720, 0x4000433ec8, 0x679704) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000243fa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x40001220c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 37 [select]: net/http.(*persistConn).readLoop(0x400000c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 38 [select]: net/http.(*persistConn).writeLoop(0x400000c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 40 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x40001220c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 53 [IO wait]: internal/poll.runtime_pollWait(0xffffaec12da0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000096198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000096180, 0x4000144000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000096180, 0x4000144000, 0x1000, 0x1000, 0x4000232a88, 0x68f7a4, 0x40001421b0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010058, 0x4000144000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c5a0, 0x4000144000, 0x1000, 0x1000, 0xf12180, 0x40001241d0, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003eb140) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003eb140, 0xa, 0x1008880, 0x4000124190, 0x199, 0x1000, 0x4000232b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003eb140, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000094300) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000094300, 0x400046e000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4001434000, 0x400046e000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4001434000, 0x400046e000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4001434040, 0x400046e000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4000142160, 0x0, 0x4000232eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4000142160, 0x0, 0x0, 0x4000232f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4000142160, 0xee0d40, 0x4000124190, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4001428000, 0x4000082480, 0x4000130780) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 32 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x4000081700, 0x4004626340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 43 [IO wait]: internal/poll.runtime_pollWait(0xffffaec12cb8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000096298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000096280, 0x400017a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000096280, 0x400017a000, 0x1000, 0x1000, 0x4000066be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000100a0, 0x400017a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000ca20, 0x400017a000, 0x1000, 0x1000, 0x4000066cf8, 0x67867c, 0x40000389c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003eb560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40003eb560, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000ca20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 44 [select]: net/http.(*persistConn).writeLoop(0x400000ca20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 82 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x4000096e80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 28 [IO wait]: internal/poll.runtime_pollWait(0xffffaec12490, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400012a798, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400012a780, 0x4002400000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400012a780, 0x4002400000, 0x1000, 0x1000, 0x4000425be0, 0x469b10, 0x4001417500) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011e088, 0x4002400000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c6480, 0x4002400000, 0x1000, 0x1000, 0x4000425cf8, 0x67867c, 0x40034609c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4003444780) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4003444780, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40001c6480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 24 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000122e80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 47 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001d9a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 67 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x400348a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 49 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x400249d230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 48 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x400010eea0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 57 [IO wait]: internal/poll.runtime_pollWait(0xffffaec122c0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4001466198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4001466180, 0x4001468000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4001466180, 0x4001468000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4001440050, 0x4001468000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000cb40, 0x4001468000, 0x1000, 0x1000, 0x4001416c00, 0x200000003, 0x4001416c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003ebce0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003ebce0, 0x400143c00a, 0x40014618c8, 0x4d3b10, 0x40001ba060, 0x400143c000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003ebce0, 0x0, 0x1, 0x0, 0x40001ba078, 0x400143c000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4003446660) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4003446660, 0x40024fd000, 0x1000, 0x1000, 0x40014619b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000080400, 0x40024fd000, 0x1000, 0x1000, 0x4000082898, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000080400, 0x40024fd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000080440, 0x40024fd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4001461dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4001461dd0, 0xa, 0x40024fd201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4001461dd0, 0x40024fd20a, 0x1, 0x4001414829, 0x0, 0x40024fd2e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4001461dd0, 0x40024a510a, 0x4001414829, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40001472f0, 0x4001432120, 0x17, 0x4001461f20, 0x4001444210, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x40001472f0, 0x113075a, 0x7, 0x4001444210, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x40001472b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 69 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x400348a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 70 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x400348a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 71 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x400348a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 95 [IO wait]: internal/poll.runtime_pollWait(0xffffaec121d8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000097418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000097400, 0x40023f9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000097400, 0x40023f9000, 0x1000, 0x1000, 0x400019f688, 0x69f178, 0x400019f6f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b8680, 0x40023f9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400027a900, 0x40023f9000, 0x1000, 0x1000, 0x5, 0x400015ef00, 0x400019f7c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40024822a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40024822a0, 0x68f90a, 0x4000245080, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40024822a0, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400041b1a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400041b1a0, 0x400248b000, 0x1000, 0x1000, 0x400019f998, 0x44e884, 0x400019f948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000122fc0, 0x400248b000, 0x1000, 0x1000, 0x4000082898, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000122fc0, 0x400248b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000123000, 0x400248b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400019fd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400019fd80, 0xa, 0x400248b201, 0x1, 0xffffaec10648, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400019fd80, 0x400248b20a, 0x1, 0x40001b5cb0, 0x0, 0x400248b259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400019fd80, 0x400246210a, 0x40001b5cb0, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40001472f0, 0x4000470f00, 0x1f, 0x400019fee0, 0x400046dad0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x40001472f0, 0x113075a, 0x7, 0x4001414840, 0xd, 0x400046dad0, 0x0, 0x2a, 0x4000053fa8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x40001472b0, 0x4001414840, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 31 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x400047aa00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 80 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4001434a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 55 [select]: net/http.(*persistConn).readLoop(0x40001c7c20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 56 [select]: net/http.(*persistConn).writeLoop(0x40001c7c20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 83 [IO wait]: internal/poll.runtime_pollWait(0xffffaec12748, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000096f98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000096f80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000096f80, 0x0, 0x4000054698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x400046d710, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x400046d710, 0x12ed9d0, 0x400046d710, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x4000096f00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 84 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40024a5040, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 85 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40024a5040, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 86 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x40024a5040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 87 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x400249def0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 30 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x400047aa00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 99 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000122b40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 90 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40000f4700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 58 [IO wait]: internal/poll.runtime_pollWait(0xffffaec123a8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4001466098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4001466080, 0x400146a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4001466080, 0x400146a000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4001440058, 0x400146a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c7e0, 0x400146a000, 0x1000, 0x1000, 0x4000102d80, 0x200000003, 0x4000102d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003ebe00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003ebe00, 0x400143c00a, 0x40014658c8, 0x4d3b10, 0x40001ba060, 0x400143c000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003ebe00, 0x0, 0x1, 0x0, 0x40001ba078, 0x400143c000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000194d50) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000194d50, 0x4001437000, 0x1000, 0x1000, 0x40014659b8, 0x0, 0x4001465988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000122840, 0x4001437000, 0x1000, 0x1000, 0x4000102a80, 0x3, 0x4001465be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000122840, 0x4001437000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000122a00, 0x4001437000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4001465dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4001465dd0, 0xa, 0x4001437201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4001465dd0, 0x400143720a, 0x1, 0x40024729bb, 0x0, 0x4001437263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4001465dd0, 0x400004620a, 0x40024729bb, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40001472f0, 0x4001430210, 0x24, 0x4001465f20, 0x40014441e0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x40001472f0, 0x113075a, 0x7, 0x40014441e0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x40001472b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 100 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40014343c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 65 [select]: net/http.(*persistConn).readLoop(0x400000c7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 98 [select]: net/http.(*persistConn).writeLoop(0x400000c7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 63 [select]: net/http.(*persistConn).readLoop(0x400000cb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 64 [select]: net/http.(*persistConn).writeLoop(0x400000cb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 29 [select]: net/http.(*persistConn).writeLoop(0x40001c6480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 114 [select]: net/http.(*persistConn).readLoop(0x400027a900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 115 [select]: net/http.(*persistConn).writeLoop(0x400027a900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 81 [IO wait]: internal/poll.runtime_pollWait(0xffffaec12660, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000209198, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000209180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000209180, 0x58ec24, 0x4000183020, 0x4000432f78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c88a50, 0x4714f4, 0x4000183020, 0x4000432f78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c88a50, 0x4001414500, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x4004449500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 130 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x4000081700, 0x400007f340, 0x400007f3b0, 0x4004626340, 0x12e8f90, 0x40046b02a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x4004843be4, 0x5, 0x4004449500, 0x12eca70, 0x4004449380, 0x130d478, 0x40001f8150, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x400348ac40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x400348ac40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 131 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40046b02a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 132 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40046b02a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 133 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x40024a7320, 0x400452df80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 134 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x4000122b00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 135 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x4004caa0a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 136 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x4004caa0a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 137 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x4000010598) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 138 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x4004caa0a0, 0x4004cc6000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 139 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400006c7d0, 0x1144559, 0x1b, 0x4002465e38, 0x4004cc6120, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004cd2180, 0x4004cc6120, 0xf1f700, 0x4000468050) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4002465f80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004527b00, 0x4004cd2180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 140 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 141 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x4000180ff0, 0xd00007, 0x4004caf648, 0x5, 0x4004cd2190, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 142 [IO wait]: internal/poll.runtime_pollWait(0xffffaec120f0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000209418, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000209400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000209400, 0xffffaec8e108, 0x30, 0x4004ca9470) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c88ff0, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c88ff0, 0x4004ca9470, 0x11ae500, 0x400348ada0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x400348ad20, 0x12ed9d0, 0x4004c88ff0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x400348ad20, 0x12ed9d0, 0x4004c88ff0, 0x4000154000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 143 [IO wait]: internal/poll.runtime_pollWait(0xffffaec12008, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000209518, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000209500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000209500, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c89008, 0xffffaec8e108, 0xffff840abcc8, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c89008, 0x665e1c, 0x0, 0xfe5e60, 0x4004cdbdc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004c89020, 0x4004ca9500, 0x11ae500, 0x400348ae80, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x400348ae00, 0x12ec710, 0x4004c89020, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x400348ae00, 0x12ec710, 0x4004c89020, 0x4000154000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 101 [IO wait]: internal/poll.runtime_pollWait(0xffffaec12578, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400012a418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400012a400, 0x4001442000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400012a400, 0x4001442000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4001440008, 0x4001442000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c7c20, 0x4001442000, 0x1000, 0x1000, 0x20, 0x30, 0x20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000082660) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000082660, 0xa, 0x0, 0x0, 0x0, 0x0, 0x40024e5c38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000082660, 0x0, 0x40024e5c78, 0x655784, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400478b110) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400478b110, 0x4004793800, 0x200, 0x200, 0x4, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40014350c0, 0x4004793800, 0x200, 0x200, 0x4001438187, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40014350c0, 0x4004793800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4001435100, 0x4004793800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40047ca9a0, 0x0, 0x40024e5eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40047ca9a0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40047ca9a0, 0xee0d40, 0x4000124820, 0x12db068, 0x4001435100) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400479abd0, 0x400147b1a0, 0x400478d3e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 144 [IO wait]: internal/poll.runtime_pollWait(0xffff845d11a8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400012b598, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400012b580, 0x4004dad000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400012b580, 0x4004dad000, 0x1000, 0x1000, 0x4004606ad8, 0x412bc0, 0x1c161d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011e408, 0x4004dad000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004d9e480, 0x4004dad000, 0x1000, 0x1000, 0x4004cdc000, 0x60, 0x58) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004daf080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004daf080, 0xa, 0x0, 0x0, 0x0, 0x0, 0x4004cdbc18) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004daf080, 0x412bc0, 0x0, 0x0, 0x4004cdbcc8, 0x412bc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004ca98c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004ca98c0, 0x4004ce0200, 0x200, 0x200, 0x4004ca94d0, 0x30, 0x4004cdbd78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004449a80, 0x4004ce0200, 0x200, 0x200, 0x0, 0xffffaec8e108, 0x4004cdbd68) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004449a80, 0x4004ce0200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004449ac0, 0x4004ce0200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004ce8420, 0x12ec700, 0x4004cdbeb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004ce8420, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004ce8420, 0xee0d40, 0x4000181180, 0x12db068, 0x4004449ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004cbe7e0, 0x4004cdc180, 0x4004cc6660) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 161 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x4002450380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 151 [IO wait]: internal/poll.runtime_pollWait(0xffff845d1378, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400012b218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400012b200, 0x4004d83000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400012b200, 0x4004d83000, 0x1000, 0x1000, 0x4004603be0, 0x469b10, 0x4000186f00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011e310, 0x4004d83000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c7b00, 0x4004d83000, 0x1000, 0x1000, 0x4004603cf8, 0x67867c, 0x400463cc00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4003445b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4003445b00, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40001c7b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 152 [select]: net/http.(*persistConn).writeLoop(0x40001c7b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 160 [semacquire]: sync.runtime_Semacquire(0x4004cff708) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004cff700) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x40000c3b80, 0x400041ac60, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffde12f969, 0x29, 0xffffde12f9cb, 0x25, 0x4004cc6d20, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004cae980, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffde12f969, 0x29, 0xffffde12f9cb, 0x25, 0x4004cc6d20, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004cae980, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004d79190, 0x4004d78e60, 0x400041ac60, 0x40003cd440, 0x4000122e80, 0x0, 0x13f, 0x400023cda8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x4000154000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004cae980, 0xb, 0x4004cae978, 0x6, 0x4004cae990, 0xd, 0x4004c874e0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x4000154000, 0x40003e0000, 0x4000110210) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x4000154000, 0x400022e990, 0x2, 0x2, 0x400022e9a0, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 162 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x4002450380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 163 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x4002450380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 164 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x4002450380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 165 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4000081e40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 177 [IO wait]: internal/poll.runtime_pollWait(0xffff845d1290, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400012b498, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400012b480, 0x4004dc9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400012b480, 0x4004dc9000, 0x1000, 0x1000, 0x4004db5be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011e410, 0x4004dc9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004d9e360, 0x4004dc9000, 0x1000, 0x1000, 0x4004db5cf8, 0x67867c, 0x4004dc62a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004daf1a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004daf1a0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004d9e360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 168 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x400467c6e0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 169 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x400467c6e0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 170 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x400467c6e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 171 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x4002450380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 172 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400006c7d0, 0x1144559, 0x1b, 0x400246be38, 0x400463d560, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004626d10, 0x400463d560, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400246bf80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004dae360, 0x4004626d10) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 178 [select]: net/http.(*persistConn).writeLoop(0x4004d9e360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 175 [select]: net/http.(*persistConn).readLoop(0x4004d9e480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 176 [select]: net/http.(*persistConn).writeLoop(0x4004d9e480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 145 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000081e80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 194 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x400467c580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 195 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x400467c580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 196 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x400467c580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 197 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x400467c580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 198 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004577c00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 199 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x4004577c00, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 200 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x4000154000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 201 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x4000154000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 202 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x4000154000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 203 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:48:10.946+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:48:10.946+00:00 [Info] Setting ipv6=false 2021-08-16T13:48:10.956+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:48:10.957+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:48:10.957+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:48:10.957+00:00 [Info] Setting log level to Info 2021-08-16T13:48:10.957+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:48:10.960+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:48:10 audit: created new audit service 2021-08-16T13:48:10.968+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:48:10.968+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:48:10.969+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:48:10.969+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:48:10.969+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:48:10.979+00:00 [Info] security context initialized 2021-08-16T13:48:10.979+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:48:10.979+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:48:10.979+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:48:10.979+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:48:10.979+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:48:10.991+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:48:10.992+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:48:10.995+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:48:11.013+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:48:11.013+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:48:11.015+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:48:11.017+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:48:11.035+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:48:11.044+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:48:11.044+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:48:11.045+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:48:11.083+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:48:11.083+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 62 us 2021-08-16T13:48:11.083+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:48:11.085+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:48:11.085+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:48:11.104+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:48:11.105+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:48:11.105+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:48:11.106+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:48:11.106+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:48:11.106+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:11.107+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:11.108+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:48:11.108+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:48:11.108+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:48:11.108+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:11.109+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:48:11.109+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:48:11.109+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:48:11.109+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:48:11.109+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:48:11.109+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:48:11.109+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:48:11.113+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 3.059546ms 2021-08-16T13:48:11.113+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:48:11.114+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:48:11.114+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:11.114+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:48:11.114+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:48:11.114+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:48:11.114+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:48:11.125+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:48:11.126+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:48:11.126+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:48:11.126+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:48:11.126+00:00 [Info] RebalanceMgr::initService Init 2021-08-16T13:48:11.126+00:00 [Info] ServiceMgr::registerWithServer nodeuuid 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:11.126+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x400245b45f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x400245b45f pc=0xbd8ea4] goroutine 111 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x400042c9f0 sp=0x400042c9c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x400042ca30 sp=0x400042c9f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x400443d040, 0x4004c09800, 0x11a9890, 0x400042cb40, 0x400443d058, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x400042cab0 sp=0x400042ca40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x400443d040, 0x4004c09800, 0x11a9890, 0x400042cb40, 0x400443d058, 0xc18f8c, 0x400042cc18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x400042caf0 sp=0x400042cab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4000032c80, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x400042cda0 sp=0x400042caf0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4000032c80, 0x4004430f50, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x400042ce60 sp=0x400042cda0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004bf05f0, 0x4004bf0600, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x400042d200 sp=0x400042ce60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004bf05f0, 0x4004bf0600, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x400042d8f0 sp=0x400042d200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004be3840, 0x4004bbfc00, 0x40003fc000, 0x4004bf05d0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x400042dfb0 sp=0x400042d8f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x400042dfb0 sp=0x400042dfb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x4000380380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x40000fbdd0, 0x4, 0x1136a87, 0xf, 0x40000f9e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 7 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400000e690) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 8 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400000e6a8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 9 [IO wait]: internal/poll.runtime_pollWait(0xffff886c5198, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0898, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0880, 0x40001a3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0880, 0x40001a3000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000105b0, 0x40001a3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x4000082cc0, 0x40001a8001, 0xdff, 0xdff, 0x400009f8e8, 0x6902d8, 0x40000f5948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400000e6f0, 0x40001a8001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40000cec60, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40000cec60, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40000cec60, 0xf5a960, 0x4000082da0, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4000082d80, 0x40000467c0, 0x40000f5b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400008c820, 0x12f8f98, 0x4000082d80, 0x4000037447, 0x16, 0x400000d680, 0x400009fc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400008c820, 0x12f8f98, 0x4000082d80, 0x400008c820, 0x4000036c58, 0x4000036c60, 0x40000f0900, 0x40000466e0, 0xee0980, 0x400000d680, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400008c820, 0x12f8f98, 0x4000082d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x40000fa3c0, 0x400009ff78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400009df78, 0x40000fa3c0, 0x12db8a8, 0x400000e6c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x40000fa3c0, 0x40000b1400, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x40000fa3c0, 0x40000b1400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 11 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 12 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 13 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 14 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 15 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 18 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x4000410000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 34 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40000537a0, 0x4, 0x4000053668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40000537a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x40000537a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x40000537a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 39 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001948e0, 0x12c18bc, 0x1, 0x400039fe58, 0x40000383c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x40004acec8, 0x40000383c0, 0x40004acec8, 0x679704) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400039ffa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x40001f8400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 37 [select]: net/http.(*persistConn).readLoop(0x40001ba360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 38 [select]: net/http.(*persistConn).writeLoop(0x40001ba360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 40 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x40001f8400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 45 [IO wait]: internal/poll.runtime_pollWait(0xffff886c50b0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0180, 0x40001a9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0180, 0x40001a9000, 0x1000, 0x1000, 0x40004c4a88, 0x68f7a4, 0x40000ce730) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000100e0, 0x40001a9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001ba360, 0x40001a9000, 0x1000, 0x1000, 0xf12180, 0x400008c400, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400033f620) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400033f620, 0xa, 0x1008880, 0x400008c3c0, 0x199, 0x1000, 0x40004c4b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400033f620, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400036d560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400036d560, 0x40004c6000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40001f8640, 0x40004c6000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40001f8640, 0x40004c6000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001f8680, 0x40004c6000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40000ce6e0, 0x0, 0x40004c4eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40000ce6e0, 0x0, 0x0, 0x40004c4f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40000ce6e0, 0xee0d40, 0x400008c3c0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40000de6c0, 0x400033fce0, 0x4000038c00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 43 [IO wait]: internal/poll.runtime_pollWait(0xffff886c4fc8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0280, 0x40004bc000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0280, 0x40004bc000, 0x1000, 0x1000, 0x400006bbe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010410, 0x40004bc000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001ba7e0, 0x40004bc000, 0x1000, 0x1000, 0x400006bcf8, 0x67867c, 0x4000038ba0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400033fb60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x400033fb60, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40001ba7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 44 [select]: net/http.(*persistConn).writeLoop(0x40001ba7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 23 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400012c580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 130 [IO wait]: internal/poll.runtime_pollWait(0xffff886c4400, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004afc118, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4004afc100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4004afc100, 0x58ec48, 0x400250c960, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004cd3500, 0x4714f4, 0x400250c960, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004cd3500, 0x400250ed00, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x4004c8c4c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 97 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001f9f00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 53 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x4000012540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 58 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400012c980) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 66 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x4000012540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 67 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x4000012540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 68 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x4000012540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 49 [select]: net/http.(*persistConn).readLoop(0x40001bb560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 87 [IO wait]: internal/poll.runtime_pollWait(0xffff886c47a0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004420298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004420280, 0x400459b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004420280, 0x400459b000, 0x1000, 0x1000, 0x4002543688, 0x69f178, 0x40025436f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400440a080, 0x400459b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400441c900, 0x400459b000, 0x1000, 0x1000, 0x5, 0x4002402500, 0x40025437c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40044249c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40044249c0, 0x68f90a, 0x4004418c60, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40044249c0, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40023f5230) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40023f5230, 0x40045f0000, 0x1000, 0x1000, 0x4002543998, 0x44e884, 0x4002543948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004414900, 0x40045f0000, 0x1000, 0x1000, 0x40004ff738, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004414900, 0x40045f0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004414940, 0x40045f0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4002543d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002543d80, 0xa, 0x40045f0201, 0x1, 0xffff884ba628, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4002543d80, 0x40045f020a, 0x1, 0x4004b01460, 0x0, 0x40045f0259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4002543d80, 0x40045c960a, 0x4004b01460, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4002500ee0, 0x4004562000, 0x1f, 0x4002543ee0, 0x4004560000, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x4002500ee0, 0x113075a, 0x7, 0x40046665f0, 0xd, 0x4004560000, 0x0, 0x0, 0x2) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x4002500ea0, 0x40046665f0, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 59 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x4000401b00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 60 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x40024aa900) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 61 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x40001ac900) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 62 [IO wait]: internal/poll.runtime_pollWait(0xffff886c4888, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001aca18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40001aca00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40001aca00, 0x0, 0x40023dfe98, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40024a5b60, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40024a5b60, 0x12ed9d0, 0x40024a5b60, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x40001ac980) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 63 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4000099c70, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 64 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4000099c70, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 65 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x4000099c70) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 82 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x40024ab3e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 46 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40001f69a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 186 [IO wait]: internal/poll.runtime_pollWait(0xffff88698f98, 0x77, 0x4000103b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000453c98, 0x77, 0x12f8600, 0x4004f60720, 0x4000453c80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitWrite(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:96 internal/poll.(*FD).WaitWrite(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:528 net.(*netFD).connect(0x4000453c80, 0x12f8668, 0x4004f60720, 0x0, 0x0, 0x12db428, 0x4004f5ba60, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:141 +0x1e4 net.(*netFD).dial(0x4000453c80, 0x12f8668, 0x4004f60720, 0x12ff2c0, 0x0, 0x12ff2c0, 0x4004f6c4b0, 0x0, 0xffffb043f6d8, 0x10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:149 +0xac net.socket(0x12f8668, 0x4004f60720, 0x112eab2, 0x3, 0x2, 0x1, 0x0, 0x0, 0x12ff2c0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:70 +0x148 net.internetSocket(0x12f8668, 0x4004f60720, 0x112eab2, 0x3, 0x12ff2c0, 0x0, 0x12ff2c0, 0x4004f6c4b0, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/ipsock_posix.go:141 +0xcc net.(*sysDialer).doDialTCP(0x4000453c00, 0x12f8668, 0x4004f60720, 0x0, 0x4004f6c4b0, 0xf74f40, 0x1c1c218, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:65 +0x8c net.(*sysDialer).dialTCP(0x4000453c00, 0x12f8668, 0x4004f60720, 0x0, 0x4004f6c4b0, 0x46e908, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:61 +0xb8 net.(*sysDialer).dialSingle(0x4000453c00, 0x12f8668, 0x4004f60720, 0x12e9300, 0x4004f6c4b0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:580 +0x4d4 net.(*sysDialer).dialSerial(0x4000453c00, 0x12f8668, 0x4004f60720, 0x4004ed9280, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:548 +0xf8 net.(*Dialer).DialContext(0x40000822a0, 0x12f8668, 0x4004f60720, 0x112eab2, 0x3, 0x4004ec3eb0, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:425 +0x550 net/http.(*Transport).dial(0x1b6e900, 0x12f8668, 0x4004f604e0, 0x112eab2, 0x3, 0x4004ec3eb0, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1162 +0x178 net/http.(*Transport).dialConn(0x1b6e900, 0x12f8668, 0x4004f604e0, 0x0, 0x4004f12930, 0x4, 0x4004ec3eb0, 0xe, 0x0, 0x4004eccc60, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1600 +0x155c net/http.(*Transport).dialConnFor(0x1b6e900, 0x4004f5e420) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1442 +0x90 created by net/http.(*Transport).queueForDial /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1411 +0x3a8 goroutine 98 [select]: net/http.(*persistConn).writeLoop(0x40001bb560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 101 [select]: net/http.(*persistConn).readLoop(0x400441c900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 102 [select]: net/http.(*persistConn).writeLoop(0x400441c900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 25 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001f8d80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 30 [IO wait]: internal/poll.runtime_pollWait(0xffff886c45d0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000452a98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000452a80, 0x400032f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000452a80, 0x400032f000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000428190, 0x400032f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001bbb00, 0x400032f000, 0x1000, 0x1000, 0x4000412900, 0x200000003, 0x4000412900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400011b560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400011b560, 0x400043800a, 0x40003a18c8, 0x4d3b10, 0x40000820c0, 0x4000438000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400011b560, 0x0, 0x1, 0x0, 0x40000820d8, 0x4000438000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400458b8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400458b8c0, 0x40004c7000, 0x1000, 0x1000, 0x40003a19b8, 0x0, 0x40003a1988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40001f9680, 0x40004c7000, 0x1000, 0x1000, 0x4000102900, 0x3, 0x40003a1be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40001f9680, 0x40004c7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001f96c0, 0x40004c7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40003a1dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003a1dd0, 0xa, 0x40004c7201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40003a1dd0, 0x40004c720a, 0x1, 0x40046665d9, 0x0, 0x40004c72e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40003a1dd0, 0x40044df70a, 0x40046665d9, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4002500ee0, 0x4000420dc8, 0x17, 0x40003a1f20, 0x400011ce10, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x4002500ee0, 0x113075a, 0x7, 0x400011ce10, 0x0, 0x4714f4, 0xdf8475800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x4002500ea0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 31 [IO wait]: internal/poll.runtime_pollWait(0xffff886c4970, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f1080, 0x40044e3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f1080, 0x40044e3000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010608, 0x40044e3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001bb560, 0x40044e3000, 0x1000, 0x1000, 0x4000412d80, 0x200000003, 0x4000412d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40004feae0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40004feae0, 0x400043800a, 0x40000a18c8, 0x4d3b10, 0x40000820c0, 0x4000438000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40004feae0, 0x0, 0x1, 0x0, 0x40000820d8, 0x4000438000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4002507aa0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4002507aa0, 0x4002574000, 0x1000, 0x1000, 0x40000a19b8, 0x44d8a8, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400046ce00, 0x4002574000, 0x1000, 0x1000, 0x400033feb8, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400046ce00, 0x4002574000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400046ce40, 0x4002574000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40000a1dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000a1dd0, 0xa, 0x4002574201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40000a1dd0, 0x400257420a, 0x1, 0x4000132b4b, 0x0, 0x4002574263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40000a1dd0, 0x400012080a, 0x4000132b4b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4002500ee0, 0x400041cba0, 0x24, 0x40000a1f20, 0x400011cdf8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x4002500ee0, 0x113075a, 0x7, 0x400011cdf8, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x4002500ea0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 120 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x4004c7a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 114 [select]: net/http.(*persistConn).readLoop(0x40001bbb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 115 [select]: net/http.(*persistConn).writeLoop(0x40001bbb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 92 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400046c340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 137 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40000e90a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 136 [semacquire]: sync.runtime_Semacquire(0x4004be3848) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004be3840) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x40003fc000, 0x400036d200, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffcc4a4969, 0x29, 0xffffcc4a49cb, 0x25, 0x4004bc2d80, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004c8bd20, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffcc4a4969, 0x29, 0xffffcc4a49cb, 0x25, 0x4004bc2d80, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004c8bd20, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x400042d190, 0x400042ce60, 0x400036d200, 0x40000b9440, 0x400012c580, 0x0, 0x17f, 0x4000178da8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x4000380380, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004c8bd20, 0xb, 0x4004c8bd18, 0x6, 0x4004c8bd30, 0xd, 0x4004cd1920, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x4000380380, 0x4000330000, 0x400036c810) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x4000380380, 0x4000415860, 0x2, 0x2, 0x4000415870, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 105 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x40023f7f00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:249 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 103 [IO wait]: internal/poll.runtime_pollWait(0xffff886c4318, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1e98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f1e80, 0x4004a7d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f1e80, 0x4004a7d000, 0x1000, 0x1000, 0x400044ba48, 0x652ce0, 0x12edac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010888, 0x4004a7d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400047efc0, 0x4004a7d000, 0x1000, 0x1000, 0x113674e, 0x412bc0, 0x4000116230) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004aa8240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004aa8240, 0xa, 0x0, 0x0, 0x0, 0x0, 0x40004a8c38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004aa8240, 0x0, 0x40004a8c78, 0x655784, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40023f5fb0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40023f5fb0, 0x4004b81800, 0x200, 0x200, 0x4, 0x0, 0x40004a8d08) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004415140, 0x4004b81800, 0x200, 0x200, 0x40023f8987, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004415140, 0x4004b81800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004415180, 0x4004b81800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004b8be40, 0x0, 0x40004a8eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004b8be40, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004b8be40, 0xee0d40, 0x400441e640, 0x12db068, 0x4004415180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004bc0750, 0x4004425980, 0x4004bc2660) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 131 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x400046d3c0, 0x400007f500, 0x400007f570, 0x4000414070, 0x12e8f90, 0x40000e89a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x4004c8ad94, 0x5, 0x4004c8c4c0, 0x12eca70, 0x4004c8c280, 0x130d478, 0x40001f68c0, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40000e8d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40000e8d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 95 [select]: net/http.(*persistConn).readLoop(0x400047efc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 96 [select]: net/http.(*persistConn).writeLoop(0x400047efc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 121 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x4004c7a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 122 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x400046d3c0, 0x4000414070) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 123 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40000e89a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 124 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40000e89a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 125 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x40044cc990, 0x4004c7c3c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 126 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x40001f8d40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 127 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x4004c7a1e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 128 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x4004c7a1e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 129 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x4000428548) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 146 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x4004c7a1e0, 0x4004c7c420) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 147 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001948e0, 0x1144559, 0x1b, 0x4002545e38, 0x4004c7c540, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004ed8100, 0x4004c7c540, 0xf1f700, 0x400011e7b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4002545f80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004c05080, 0x4004ed8100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 148 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 149 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x4000404b90, 0xd00007, 0x4004ec25c8, 0x5, 0x4004ed8110, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 150 [IO wait]: internal/poll.runtime_pollWait(0xffff886c46b8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000453298, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000453280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000453280, 0xffffb05a75b8, 0x30, 0x4004bea240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004eb95d8, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004eb95d8, 0x4004bea240, 0x11ae500, 0x4000464940, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40004648c0, 0x12ed9d0, 0x4004eb95d8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40004648c0, 0x12ed9d0, 0x4004eb95d8, 0x4000380380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 151 [IO wait]: internal/poll.runtime_pollWait(0xffff886c4a58, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000453398, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000453380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000453380, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004eb95f0, 0xffffb05a7108, 0xffff881c86f8, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004eb95f0, 0x665e1c, 0x0, 0xfe5e60, 0x4004b13dc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004eb9608, 0x4004cdb1d0, 0x11ae500, 0x4000464a20, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40004649a0, 0x12ec710, 0x4004eb9608, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40004649a0, 0x12ec710, 0x4004eb9608, 0x4000380380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 179 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x4004aaf080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:662 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 180 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x4004aaf080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:752 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 157 [IO wait]: internal/poll.runtime_pollWait(0xffff88699250, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004afc618, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004afc600, 0x4004e9f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004afc600, 0x4004e9f000, 0x1000, 0x1000, 0x4004d90be0, 0x469b10, 0x4004ed5500) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40004285c0, 0x4004e9f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004ecc5a0, 0x4004e9f000, 0x1000, 0x1000, 0x4004d90cf8, 0x67867c, 0x4004c7ca80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004c05860) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004c05860, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004ecc5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 178 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x4004aaf080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:612 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 104 [IO wait]: internal/poll.runtime_pollWait(0xffff88699168, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004afc898, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004afc880, 0x4004e0f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004afc880, 0x4004e0f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010b40, 0x4004e0f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400047fd40, 0x4004e0f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004e6e1e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004e6e1e0, 0xa, 0x40004a6be8, 0x48731c, 0x9, 0x1, 0x49) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004e6e1e0, 0x0, 0x40004a6c48, 0x57d438, 0x11320c7, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004bea690) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004bea690, 0x4004bf4800, 0x200, 0x200, 0x4004ec2638, 0x4004ec2638, 0x400041d7a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004415500, 0x4004bf4800, 0x200, 0x200, 0x84feb518ef, 0x65a85b2, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004415500, 0x4004bf4800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004415540, 0x4004bf4800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004beedc0, 0x0, 0x40004a6eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004beedc0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004beedc0, 0xee0d40, 0x400441e870, 0x12db068, 0x4004415540) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004bc0cf0, 0x4004425e00, 0x4004bc2c00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 158 [select]: net/http.(*persistConn).writeLoop(0x4004ecc5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 161 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x4004aaf080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:224 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 160 [select]: net/http.(*persistConn).roundTrip(0x400047fc20, 0x4004f322c0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004edb400, 0xf8, 0x10a8660, 0x4000174e18) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004edb400, 0x1b6e900, 0xc03eb924c7872777, 0x1bfbb2b435) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004edb300, 0x12db048, 0x1b6e900, 0xc03eb924c7872777, 0x1bfbb2b435, 0x1be4a80, 0x0, 0x40004286c8, 0x4004f6c090, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004f6c060, 0x4004edb300, 0xc03eb924c7872777, 0x1bfbb2b435, 0x1be4a80, 0x40004286c8, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004f6c060, 0x4004edb300, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004f126f0, 0x1b, 0x4000175580, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004f22750, 0x112f905, 0x6, 0x12db908, 0x4000121580, 0xee40c0, 0x4000453a18, 0x4000175580, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x4004480580, 0x46, 0x12db908, 0x4000121580, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x4004480580, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x20, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000175d40, 0x4, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x4000136900, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x4000136900, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4004c8cb80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1267 +0xf8 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 138 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40000e90a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 139 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40000e90a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 140 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40000e90a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 141 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4004c8cb40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 169 [IO wait]: internal/poll.runtime_pollWait(0xffff886c44e8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004afc798, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004afc780, 0x4004e71000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004afc780, 0x4004e71000, 0x1000, 0x1000, 0x4004e57be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010b48, 0x4004e71000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400047fc20, 0x4004e71000, 0x1000, 0x1000, 0x4004e57cf8, 0x67867c, 0x4004e07740) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004e6e300) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004e6e300, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400047fc20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 144 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004aaf1e0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 145 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004aaf1e0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 162 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4004aaf1e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 163 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40000e90a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 164 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001948e0, 0x1144559, 0x1b, 0x4002541e38, 0x4004e06a20, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004aff280, 0x4004e06a20, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4002541f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004aa94a0, 0x4004aff280) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 170 [select]: net/http.(*persistConn).writeLoop(0x400047fc20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 167 [select]: net/http.(*persistConn).readLoop(0x400047fd40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 168 [select]: net/http.(*persistConn).writeLoop(0x400047fd40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 107 [select]: net/http.(*Transport).getConn(0x1b6e900, 0x4004f32300, 0x0, 0x4004f12840, 0x4, 0x4004ec3e50, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1368 +0x418 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004edb600, 0xf8, 0x10a8660, 0x4004f06dd8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:579 +0x5c4 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004edb600, 0x1b6e900, 0xc03eb924c7887bb8, 0x1bfbb4086e) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004edb500, 0x12db048, 0x1b6e900, 0xc03eb924c7887bb8, 0x1bfbb4086e, 0x1be4a80, 0x0, 0x40004286d8, 0x4004f6c240, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004f6c210, 0x4004edb500, 0xc03eb924c7887bb8, 0x1bfbb4086e, 0x1be4a80, 0x40004286d8, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004f6c210, 0x4004edb500, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004f12810, 0x1b, 0x4004f07540, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004f22a20, 0x112f905, 0x6, 0x12db908, 0x40001215e0, 0xee40c0, 0x4000453a98, 0x4004f07540, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x400442a680, 0x46, 0x12db908, 0x40001215e0, 0x4004e5a598, 0x81ee08, 0x4004e5a748, 0x95dcbc, 0x4004480580, 0x46, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x400442a680, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x240, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004f07d00, 0x4004f60240, 0xffffb043e088) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x4004f70000, 0xd47600, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x4004f70000, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x4000380380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9702 +0x344 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 108 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x4000380380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1090 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 109 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x4000380380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1017 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 110 [sleep]: time.Sleep(0x12a05f200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/plasma.RunMemQuotaTunerIteration(0x40023e9fb0, 0xdf74d8, 0x4000038120) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:211 +0x24c github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:247 +0xc0 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 goroutine 181 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).processSchedIndexes(0x4004aaf080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:296 +0x44 created by github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:218 +0x3c goroutine 182 [IO wait]: internal/poll.runtime_pollWait(0xffff88699080, 0x77, 0x4004f6e180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000453b98, 0x77, 0x12f8600, 0x4004f60480, 0x4000453b80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitWrite(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:96 internal/poll.(*FD).WaitWrite(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:528 net.(*netFD).connect(0x4000453b80, 0x12f8668, 0x4004f60480, 0x0, 0x0, 0x12db428, 0x4004f5ba00, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:141 +0x1e4 net.(*netFD).dial(0x4000453b80, 0x12f8668, 0x4004f60480, 0x12ff2c0, 0x0, 0x12ff2c0, 0x4004f6c330, 0x0, 0xffff882f8a00, 0xffff8847cfff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:149 +0xac net.socket(0x12f8668, 0x4004f60480, 0x112eab2, 0x3, 0x2, 0x1, 0x0, 0x0, 0x12ff2c0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:70 +0x148 net.internetSocket(0x12f8668, 0x4004f60480, 0x112eab2, 0x3, 0x12ff2c0, 0x0, 0x12ff2c0, 0x4004f6c330, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/ipsock_posix.go:141 +0xcc net.(*sysDialer).doDialTCP(0x4000453b00, 0x12f8668, 0x4004f60480, 0x0, 0x4004f6c330, 0xf74f40, 0x1c1c218, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:65 +0x8c net.(*sysDialer).dialTCP(0x4000453b00, 0x12f8668, 0x4004f60480, 0x0, 0x4004f6c330, 0x46e908, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:61 +0xb8 net.(*sysDialer).dialSingle(0x4000453b00, 0x12f8668, 0x4004f60480, 0x12e9300, 0x4004f6c330, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:580 +0x4d4 net.(*sysDialer).dialSerial(0x4000453b00, 0x12f8668, 0x4004f60480, 0x4004ed9170, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:548 +0xf8 net.(*Dialer).DialContext(0x40000822a0, 0x12f8668, 0x4004f60480, 0x112eab2, 0x3, 0x4004ec3e50, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:425 +0x550 net/http.(*Transport).dial(0x1b6e900, 0x12f8668, 0x4004f602a0, 0x112eab2, 0x3, 0x4004ec3e50, 0xe, 0x0, 0x4000124480, 0x2, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1162 +0x178 net/http.(*Transport).dialConn(0x1b6e900, 0x12f8668, 0x4004f602a0, 0x0, 0x4004f12840, 0x4, 0x4004ec3e50, 0xe, 0x0, 0x4004ecca20, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1600 +0x155c net/http.(*Transport).dialConnFor(0x1b6e900, 0x4004f5e370) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1442 +0x90 created by net/http.(*Transport).queueForDial /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1411 +0x3a8 goroutine 183 [select]: net.(*netFD).connect.func2(0x12f8668, 0x4004f60480, 0x4000453b80, 0x4004f720c0, 0x4004f72060) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:118 +0x70 created by net.(*netFD).connect /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:117 +0x1c0 goroutine 184 [IO wait]: internal/poll.runtime_pollWait(0xffff88698eb0, 0x77, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000453d98, 0x77, 0x4000036000, 0x1, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitWrite(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:96 internal/poll.(*FD).WaitWrite(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:528 net.(*netFD).connect(0x4000453d80, 0x12f8630, 0x4000036098, 0x0, 0x0, 0x12db428, 0x4004f5baa0, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:141 +0x1e4 net.(*netFD).dial(0x4000453d80, 0x12f8630, 0x4000036098, 0x12ff2c0, 0x0, 0x12ff2c0, 0x4004f6c5a0, 0x0, 0xffffb043f6d8, 0x10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:149 +0xac net.socket(0x12f8630, 0x4000036098, 0x112eab2, 0x3, 0x2, 0x1, 0x0, 0x0, 0x12ff2c0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:70 +0x148 net.internetSocket(0x12f8630, 0x4000036098, 0x112eab2, 0x3, 0x12ff2c0, 0x0, 0x12ff2c0, 0x4004f6c5a0, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/ipsock_posix.go:141 +0xcc net.(*sysDialer).doDialTCP(0x4000453d00, 0x12f8630, 0x4000036098, 0x0, 0x4004f6c5a0, 0x0, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:65 +0x8c net.(*sysDialer).dialTCP(0x4000453d00, 0x12f8630, 0x4000036098, 0x0, 0x4004f6c5a0, 0xf89d40, 0x4004f723ec, 0x1be4a80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:61 +0xb8 net.(*sysDialer).dialSingle(0x4000453d00, 0x12f8630, 0x4000036098, 0x12e9300, 0x4004f6c5a0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:580 +0x4d4 net.(*sysDialer).dialSerial(0x4000453d00, 0x12f8630, 0x4000036098, 0x4004ed92c0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:548 +0xf8 net.(*Dialer).DialContext(0x4005083cb8, 0x12f8630, 0x4000036098, 0x112eab2, 0x3, 0x4004f723ec, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:425 +0x550 net.(*Dialer).Dial(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:348 net.Dial(0x112eab2, 0x3, 0x4004f723ec, 0xe, 0x30, 0x1008940, 0x1, 0x4004f6c540) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:319 +0x7c github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4004f6c540, 0x4005083f10, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:116 +0xc0 github.com/couchbase/cbauth/revrpc.BabysitService(0x4005083f10, 0x4004f6c540, 0x0, 0x0, 0x0, 0xd8d03c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth/service.RegisterManager(0x13007d8, 0x40023f7f00, 0x0, 0x0, 0x4004e5af80, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:146 +0xa0 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).registerWithServer(0x40023f7f00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:301 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:205 +0x78 goroutine 185 [select]: net/http.(*Transport).getConn(0x1b6e900, 0x4004f32340, 0x0, 0x4004f12930, 0x4, 0x4004ec3eb0, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1368 +0x418 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004edb800, 0xf8, 0x10a8660, 0x4004e5a798) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:579 +0x5c4 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004edb800, 0x1b6e900, 0xc03eba32c78a3ece, 0x11770b9fb8c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004edb700, 0x12db048, 0x1b6e900, 0xc03eba32c78a3ece, 0x11770b9fb8c, 0x1be4a80, 0x0, 0x40004286f0, 0x4004f6c3c0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004f6c390, 0x4004edb700, 0xc03eba32c78a3ece, 0x11770b9fb8c, 0x1be4a80, 0x40004286f0, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004f6c390, 0x4004edb700, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004f5ba40, 0x1f, 0x4004e5ad10, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/indexer.getWithAuth(0x4004f5ba40, 0x1f, 0xe, 0x113846e, 0x11) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:3077 +0x68 github.com/couchbase/indexing/secondary/indexer.getGlobalTopology(0x4004be2e60, 0xe, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:2266 +0x78 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).updateNodeList(0x40023f7f00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:220 +0x3c created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:206 +0x98 goroutine 187 [select]: net.(*netFD).connect.func2(0x12f8668, 0x4004f60720, 0x4000453c80, 0x4004f72360, 0x4004f72300) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:118 +0x70 created by net.(*netFD).connect /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:117 +0x1c0 Initializing write barrier = 8000 2021-08-16T13:48:16.335+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:48:16.335+00:00 [Info] Setting ipv6=false 2021-08-16T13:48:16.350+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:48:16.351+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:48:16.351+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:48:16.352+00:00 [Info] Setting log level to Info 2021-08-16T13:48:16.352+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:48:16.355+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:48:16 audit: created new audit service 2021-08-16T13:48:16.361+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:48:16.361+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:48:16.361+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:48:16.361+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:48:16.361+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:48:16.371+00:00 [Info] security context initialized 2021-08-16T13:48:16.371+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:48:16.372+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:48:16.372+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:48:16.372+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:48:16.372+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:48:16.384+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:48:16.385+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:48:16.398+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:48:16.413+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:48:16.413+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:48:16.415+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:48:16.415+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:48:16.416+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:48:16.418+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:48:16.419+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:48:16.420+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:48:16.466+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:48:16.466+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 49 us 2021-08-16T13:48:16.466+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:48:16.468+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:48:16.469+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:48:16.469+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:48:16.469+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:48:16.469+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:48:16.469+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:48:16.469+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:48:16.470+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:16.471+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:16.472+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:48:16.472+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:48:16.472+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:48:16.472+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:16.472+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:48:16.472+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:48:16.472+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:48:16.472+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:48:16.472+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:48:16.472+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:48:16.472+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:48:16.478+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 5.515364ms 2021-08-16T13:48:16.478+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:48:16.480+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:48:16.480+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:16.480+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:48:16.480+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:48:16.480+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:48:16.480+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:48:16.490+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:48:16.490+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:48:16.491+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:48:16.491+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards unexpected fault address 2021-08-16T13:48:16.491+00:00 [Info] : Recover shard shard1 from metadata completed. 2021-08-16T13:48:16.491+00:00 [Info] RebalanceMgr::initService Init 2021-08-16T13:48:16.491+00:00 [Info] ServiceMgr::registerWithServer nodeuuid 8a95cde683482bdec453edbb8b18eb5d 0x40045fe91f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x40045fe91f pc=0xbd8ea4] goroutine 180 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x400465e9f0 sp=0x400465e9c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x400465ea30 sp=0x400465e9f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x40003d7860, 0x4004eaa000, 0x11a9890, 0x400465eb40, 0x40003d7878, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x400465eab0 sp=0x400465ea40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x40003d7860, 0x4004eaa000, 0x11a9890, 0x400465eb40, 0x40003d7878, 0xc18f8c, 0x400465ec18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x400465eaf0 sp=0x400465eab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4004510640, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x400465eda0 sp=0x400465eaf0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4004510640, 0x40014675e0, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x400465ee60 sp=0x400465eda0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x40045fdb70, 0x40045fdb80, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x400465f200 sp=0x400465ee60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x40045fdb70, 0x40045fdb80, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x400465f8f0 sp=0x400465f200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004dbfde0, 0x4004e88700, 0x400458cb00, 0x40045fdb50) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x400465ffb0 sp=0x400465f8f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x400465ffb0 sp=0x400465ffb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x4000132000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x40000fcf00, 0x4, 0x1136a87, 0xf, 0x400018be68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 18 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400010c100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 7 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400000e648) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 8 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400000e660) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 9 [IO wait]: internal/poll.runtime_pollWait(0xffff5c42ce88, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400019c118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400019c100, 0x40001c1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400019c100, 0x40001c1000, 0x1000, 0x1000, 0x1311860, 0x47b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000190030, 0x40001c1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x40001802a0, 0x400144c001, 0xdff, 0xdff, 0x400009f8e8, 0x6902d8, 0x4000187948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400018c048, 0x400144c001, 0xdff, 0xdff, 0x40001879f8, 0x6f1968, 0x4000187938) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x4001444000, 0xa, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4001444000, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4001444000, 0xf5a960, 0x4001414140, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4001414120, 0x40014460e0, 0x4000187b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x4001434000, 0x12f8f98, 0x4001414120, 0x4001430857, 0x16, 0x4001450120, 0x400009fc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x4001434000, 0x12f8f98, 0x4001414120, 0x4001434000, 0x4001430060, 0x4001430070, 0x4001442000, 0x4001446000, 0xee0980, 0x4001450120, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x4001434000, 0x12f8f98, 0x4001414120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4000097e60, 0x400009ff78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400009bf78, 0x4000097e60, 0x12db8a8, 0x400018c018, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x4000097e60, 0x40000b12c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x4000097e60, 0x40000b12c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 10 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 11 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 12 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 13 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 14 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 15 [syscall]: syscall.Syscall(0x3f, 0x0, 0x4000054fa0, 0x4, 0x4000054e68, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x4000054fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x4000054fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x4000054fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 37 [IO wait]: internal/poll.runtime_pollWait(0xffff5c42cda0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0080, 0x40001c0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0080, 0x40001c0000, 0x1000, 0x1000, 0x400006aa88, 0x68f7a4, 0x4001470730) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010008, 0x40001c0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003d25a0, 0x40001c0000, 0x1000, 0x1000, 0xf12180, 0x40001a20e0, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000180360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000180360, 0xa, 0x1008880, 0x40001a20a0, 0x199, 0xad9, 0x400006ab00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000180360, 0x2, 0x0, 0x9, 0x56b, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000097f50) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000097f50, 0x400147e000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000081640, 0x400147e000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000081640, 0x400147e000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000081680, 0x400147e000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40014706e0, 0x0, 0x400006aeb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40014706e0, 0x0, 0x0, 0x400006af70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40014706e0, 0xee0d40, 0x40001a20a0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400016c510, 0x400010e660, 0x400147c180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 54 [select]: net/http.(*persistConn).writeLoop(0x40003d2c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 51 [select]: net/http.(*persistConn).readLoop(0x40003d25a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 52 [select]: net/http.(*persistConn).writeLoop(0x40003d25a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 22 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40002142d0, 0x12c18bc, 0x1, 0x40000a1e58, 0x40001306c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x400141aec8, 0x40001306c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40000a1fa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4000118080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 23 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4000118080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 101 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4001436580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 53 [IO wait]: internal/poll.runtime_pollWait(0xffff5c42ccb8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0180, 0x40001d6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0180, 0x40001d6000, 0x1000, 0x1000, 0x40001a6be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000100b8, 0x40001d6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003d2c60, 0x40001d6000, 0x1000, 0x1000, 0x40001a6cf8, 0x67867c, 0x4000130960) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000180480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000180480, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40003d2c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 16 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000118940) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 40 [IO wait]: internal/poll.runtime_pollWait(0xffff5c42c748, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4001442798, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4001442780, 0x4000144000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4001442780, 0x4000144000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400143c0a0, 0x4000144000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40014505a0, 0x4000144000, 0x1000, 0x1000, 0x4001416600, 0x200000003, 0x4001416600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400010eba0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400010eba0, 0x40001ae00a, 0x40004778c8, 0x4d3b10, 0x40000820c0, 0x40001ae000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400010eba0, 0x0, 0x1, 0x0, 0x40000820d8, 0x40001ae000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000182c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000182c60, 0x4002483000, 0x1000, 0x1000, 0x40004779b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000118c40, 0x4002483000, 0x1000, 0x1000, 0x400010e838, 0x45b65c, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000118c40, 0x4002483000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000118c80, 0x4002483000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000477dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000477dd0, 0xa, 0x4002483201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000477dd0, 0x400248320a, 0x1, 0x400011a9e9, 0x0, 0x40024832e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000477dd0, 0x400013a80a, 0x400011a9e9, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400013a6c0, 0x40014325e8, 0x17, 0x4000477f20, 0x400143e648, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x400013a6c0, 0x113075a, 0x7, 0x400143e648, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x400013a680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 46 [IO wait]: internal/poll.runtime_pollWait(0xffff5c42c1d8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011cb18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011cb00, 0x4002502000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011cb00, 0x4002502000, 0x1000, 0x1000, 0x4000479688, 0x69f178, 0x40004796f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000122188, 0x4002502000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000172c60, 0x4002502000, 0x1000, 0x1000, 0x5, 0x40024d4a00, 0x40004797c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400249c2a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400249c2a0, 0x68f90a, 0x40001da840, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400249c2a0, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40001b4870) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40001b4870, 0x400014f000, 0x1000, 0x1000, 0x4000479998, 0x44e884, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40000819c0, 0x400014f000, 0x1000, 0x1000, 0x400010e838, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40000819c0, 0x400014f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000081a00, 0x400014f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000479d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000479d80, 0xa, 0x400014f201, 0x1, 0xffff5c47caa8, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000479d80, 0x400014f20a, 0x1, 0x4002527200, 0x0, 0x400014f259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000479d80, 0x400447250a, 0x4002527200, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400013a6c0, 0x40001344e0, 0x1f, 0x4000479ee0, 0x400143e978, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x400013a6c0, 0x113075a, 0x7, 0x400011aa00, 0xd, 0x400143e978, 0x0, 0x2a, 0x400141efa8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x400013a680, 0x400011aa00, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 41 [IO wait]: internal/poll.runtime_pollWait(0xffff5c42c490, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011c718, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011c700, 0x400047e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011c700, 0x400047e000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000122120, 0x400047e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000172900, 0x400047e000, 0x1000, 0x1000, 0x40001caa80, 0x200000003, 0x40001caa80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4001415e60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4001415e60, 0x40001ae00a, 0x400017b8c8, 0x4d3b10, 0x40000820c0, 0x40001ae000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4001415e60, 0x0, 0x1, 0x0, 0x40000820d8, 0x40001ae000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000182840) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000182840, 0x4002482000, 0x1000, 0x1000, 0x400017b9b8, 0x0, 0x400017b988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000118b00, 0x4002482000, 0x1000, 0x1000, 0x4001416480, 0x3, 0x400017bbe0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000118b00, 0x4002482000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000118b40, 0x4002482000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400017bdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400017bdd0, 0xa, 0x4002482201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400017bdd0, 0x400248220a, 0x1, 0x4001430bdb, 0x0, 0x4002482263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400017bdd0, 0x400144660a, 0x4001430bdb, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400013a6c0, 0x4000112330, 0x24, 0x400017bf20, 0x400143e630, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x400013a6c0, 0x113075a, 0x7, 0x400143e630, 0x0, 0xf4240, 0x40003a1080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x400013a680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 67 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x4002400620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 44 [select]: net/http.(*persistConn).readLoop(0x40014505a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 26 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x4002400620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 27 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x4002400620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 28 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x4002400620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 123 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x40046acdc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 122 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4001437080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 30 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000118e40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 31 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x40014149c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 32 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x400246b0b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 33 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x400011c480) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 82 [IO wait]: internal/poll.runtime_pollWait(0xffff5c42c660, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011c598, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400011c580, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400011c580, 0x0, 0x4001420698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4000467308, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4000467308, 0x12ed9d0, 0x4000467308, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x400011c500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 83 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4000099a00, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 84 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4000099a00, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 85 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x4000099a00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 86 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x400246bb90) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 124 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x40046acdc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 94 [select]: net/http.(*persistConn).readLoop(0x4000172900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 89 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40002727e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 95 [select]: net/http.(*persistConn).writeLoop(0x4000172900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 102 [IO wait]: internal/poll.runtime_pollWait(0xffff5c42c0f0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011cc18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011cc00, 0x4002500000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011cc00, 0x4002500000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000122178, 0x4002500000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000173200, 0x4002500000, 0x1000, 0x1000, 0x20, 0x30, 0x20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400249c240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400249c240, 0xa, 0x0, 0x0, 0x0, 0x0, 0x4000055438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400249c240, 0x7356e4, 0xffffe7eb094e, 0xe, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40044b5950) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40044b5950, 0x4004be4200, 0x200, 0x200, 0xffffe7eb094e, 0xe, 0x4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004bde1c0, 0x4004be4200, 0x200, 0x200, 0x400143acc7, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004bde1c0, 0x4004be4200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004bde200, 0x4004be4200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004be8160, 0x4001416e00, 0x40000556b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004be8160, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004be8160, 0xee0d40, 0x4000120b90, 0x12db068, 0x4004bde200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004758990, 0x40044991a0, 0x4004be0300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 119 [select]: net/http.(*persistConn).writeLoop(0x4000173200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 45 [select]: net/http.(*persistConn).writeLoop(0x40014505a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 116 [select]: net/http.(*persistConn).readLoop(0x40024925a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 120 [select]: net/http.(*persistConn).readLoop(0x4000172c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 121 [select]: net/http.(*persistConn).writeLoop(0x4000172c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 118 [select]: net/http.(*persistConn).readLoop(0x4000173200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 117 [select]: net/http.(*persistConn).writeLoop(0x40024925a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 47 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000119dc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 48 [IO wait]: internal/poll.runtime_pollWait(0xffff5c42c578, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4001442e18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4001442e00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4001442e00, 0x12d9d48, 0x4000118f40, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40045ce0a8, 0x0, 0x4002497320, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40045ce0a8, 0x40023cd600, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x40045028c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 49 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x40047cc640, 0x4001466a10, 0x4001466a80, 0x4004c3e890, 0x12e8f90, 0x4002400a80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x400459d614, 0x5, 0x40045028c0, 0x12eca70, 0x4004502780, 0x130d478, 0x4000170310, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x400450e460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x400450e460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 130 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x4002400a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 131 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x4002400a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 132 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x4002494b40, 0x40045ca420) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 133 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x4001436540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 134 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x40001248c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 135 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x40001248c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 136 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x400143c5c8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 137 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x40001248c0, 0x40045ca480) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 125 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x40047cc640, 0x4004c3e890) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 138 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40002142d0, 0x1144559, 0x1b, 0x4002475e38, 0x40045ca5a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x40045fc8b0, 0x40045ca5a0, 0xf1f700, 0x4000215b50) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4002475f80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004cfa060, 0x40045fc8b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 139 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 140 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x4001434aa0, 0xd00007, 0x40045f6fc8, 0x5, 0x40045fc8c0, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 141 [IO wait]: internal/poll.runtime_pollWait(0xffff5c42c3a8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4001442f98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4001442f80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4001442f80, 0x0, 0x40003b4d88, 0x665fe4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40045ce648, 0x611a6ca0, 0x40003b4db8, 0x445c24) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40045ce648, 0x40003b4e20, 0x40003b4e28, 0x18, 0x40045e7380) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x400450e540, 0x12ed9d0, 0x40045ce648, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x400450e540, 0x12ed9d0, 0x40045ce648, 0x4000132000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 142 [IO wait]: internal/poll.runtime_pollWait(0xffff5c42c008, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4001443098, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4001443080, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4001443080, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40045ce660, 0xffff8576c108, 0xffff5c46cbb8, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40045ce660, 0x665e1c, 0x0, 0xfe5e60, 0x400242cdc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x40045ce678, 0x40001118c0, 0x11ae500, 0x400450e6a0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x400450e620, 0x12ec710, 0x40045ce678, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x400450e620, 0x12ec710, 0x40045ce678, 0x4000132000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 128 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x4002400d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 144 [IO wait]: internal/poll.runtime_pollWait(0xffff5c448768, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4001443218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4001443200, 0x4004cfc000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4001443200, 0x4004cfc000, 0x1000, 0x1000, 0x4004d05be0, 0x469b10, 0x4001417680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400143c600, 0x4004cfc000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40024939e0, 0x4004cfc000, 0x1000, 0x1000, 0x4004d05cf8, 0x67867c, 0x40045ca8a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004cfa300) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004cfa300, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40024939e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 109 [select]: net/http.(*persistConn).writeLoop(0x400000d680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 145 [select]: net/http.(*persistConn).writeLoop(0x40024939e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 127 [semacquire]: sync.runtime_Semacquire(0x4004dbfde8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004dbfde0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x400458cb00, 0x40000fd7a0, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffe7eb0969, 0x29, 0xffffe7eb09cb, 0x25, 0x4004e84180, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x40045f6300, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffe7eb0969, 0x29, 0xffffe7eb09cb, 0x25, 0x4004e84180, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x40045f6300, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x400465f190, 0x400465ee60, 0x40000fd7a0, 0x4002468000, 0x4000118940, 0x0, 0x4124ec, 0x1c211e8, 0xffff5c2c72f8, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x4000132000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x40045f6300, 0xb, 0x40045f62f8, 0x6, 0x40045f6310, 0xd, 0x40045c2e40, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x4000132000, 0x40003a0000, 0x40001100c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x4000132000, 0x40014390b0, 0x2, 0x2, 0x40014390c0, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 129 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x4002400d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 162 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x4002400d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 163 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x4002400d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 164 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x40047cc780) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 108 [IO wait]: internal/poll.runtime_pollWait(0xffff5c448680, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011d698, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011d680, 0x4004d39000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011d680, 0x4004d39000, 0x1000, 0x1000, 0x40003b0be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010950, 0x4004d39000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000d680, 0x4004d39000, 0x1000, 0x1000, 0x40003b0cf8, 0x67867c, 0x4004be0de0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004499c80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004499c80, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000d680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 167 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40047a4f20, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 168 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40047a4f20, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 169 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x40047a4f20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 170 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x4002400d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 171 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40002142d0, 0x1144559, 0x1b, 0x4000177e38, 0x4004c70180, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004c3eaa0, 0x4004c70180, 0x2, 0x4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x2, 0x4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000177f80, 0x400250e100, 0x2) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004c6e360, 0x4004c3eaa0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 176 [IO wait]: internal/poll.runtime_pollWait(0xffff5c448598, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011d798, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011d780, 0x4004c4d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011d780, 0x4004c4d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000122418, 0x4004c4d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000d8c0, 0x4004c4d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004c6f080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004c6f080, 0xa, 0x0, 0x0, 0x0, 0x0, 0x4000055438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004c6f080, 0x41ab8c, 0xffff4cc5a400, 0x100100000000, 0xffff4cde9fff, 0x400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40047b3c20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40047b3c20, 0x40047ff200, 0x200, 0x200, 0x4000055508, 0x41cff4, 0xffff4cc5a400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40047cc8c0, 0x40047ff200, 0x200, 0x200, 0xffff5c494e20, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40047cc8c0, 0x40047ff200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40047cc900, 0x40047ff200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004e08000, 0x4001416e00, 0x40000556b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004e08000, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004e08000, 0xee0d40, 0x40001a3130, 0x12db068, 0x40047cc900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40047b1dd0, 0x4004c6f2c0, 0x4004c70de0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 174 [select]: net/http.(*persistConn).readLoop(0x400000d8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 175 [select]: net/http.(*persistConn).writeLoop(0x400000d8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 153 [select]: net/http.(*persistConn).roundTrip(0x400000d680, 0x40047ccc80, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004c40e00, 0xf8, 0x10a8660, 0x400017ce18) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004c40e00, 0x1b6e900, 0xc03eb9261d4954e2, 0x1bfa29683e) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004c40d00, 0x12db048, 0x1b6e900, 0xc03eb9261d4954e2, 0x1bfa29683e, 0x1be4a80, 0x0, 0x4000122460, 0x4004e36390, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004e36360, 0x4004c40d00, 0xc03eb9261d4954e2, 0x1bfa29683e, 0x1be4a80, 0x4000122460, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004e36360, 0x4004c40d00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x40047a1860, 0x1b, 0x400017d580, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004e12750, 0x112f905, 0x6, 0x12db908, 0x40014477e0, 0xee40c0, 0x400011d918, 0x400017d580, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x400003e480, 0x46, 0x12db908, 0x40014477e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x400003e480, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x20, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400017dd40, 0x4, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x40000b98c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x40000b98c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40047cc7c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1267 +0xf8 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 154 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x40047a4dc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:224 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 155 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x40047a4dc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:612 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 156 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x40047a4dc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:662 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 157 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x40047a4dc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:752 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 158 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004dc2700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:249 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 201 [IO wait]: internal/poll.runtime_pollWait(0xffff5c4482e0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011dc18, 0x72, 0x0, 0x1, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011dc00, 0x4004e369d1, 0x1, 0x1, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011dc00, 0x4004e369d1, 0x1, 0x1, 0x4004c77701, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000122498, 0x4004e369d1, 0x1, 0x1, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*connReader).backgroundRead(0x4004e369c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:692 +0x50 created by net/http.(*connReader).startBackgroundRead /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:688 +0xc4 goroutine 160 [runnable]: net/http.(*persistConn).roundTrip(0x40024925a0, 0x40047ccd00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004c41200, 0xf8, 0x10a8660, 0x4004d08dd8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004c41200, 0x1b6e900, 0xc03eb9261d4c8a7b, 0x1bfa2c9dd7) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004c41100, 0x12db048, 0x1b6e900, 0xc03eb9261d4c8a7b, 0x1bfa2c9dd7, 0x1be4a80, 0x0, 0x4000122488, 0x4004e366c0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004e36690, 0x4004c41100, 0xc03eb9261d4c8a7b, 0x1bfa2c9dd7, 0x1be4a80, 0x4000122488, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004e36690, 0x4004c41100, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x40047a1a70, 0x1b, 0x4004d09540, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004e12c60, 0x112f905, 0x6, 0x12db908, 0x4001447860, 0xee40c0, 0x400011da98, 0x4004d09540, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x400003e580, 0x46, 0x12db908, 0x4001447860, 0x12f8668, 0x4004c41000, 0x4004c3f340, 0x40047a1980, 0xffff5c4957b0, 0x10, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x400003e580, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x240, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004d09d00, 0x4004e365d0, 0xffff5c2c94f8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x4004e3e240, 0xd47600, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x4004e3e240, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x4000132000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9702 +0x344 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 161 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x4000132000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1090 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 178 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x4000132000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1017 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 179 [sleep]: time.Sleep(0x12a05f200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/plasma.RunMemQuotaTunerIteration(0x4004e75fb0, 0xdf74d8, 0x4000130540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:211 +0x24c github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:247 +0xc0 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 goroutine 177 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).processSchedIndexes(0x40047a4dc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:296 +0x44 created by github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:218 +0x3c goroutine 194 [IO wait]: internal/poll.runtime_pollWait(0xffff5c4483c8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011db98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011db80, 0x4004e58000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011db80, 0x4004e58000, 0x1000, 0x1000, 0x0, 0x4004e19000, 0x4004e4fa38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000122490, 0x4004e58000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).fill(0x4004e56180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004e56180, 0x40a, 0x4004e58000, 0x0, 0xffff00001001, 0x400, 0x400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).ReadLine(0x4004e56180, 0x1000, 0xffff5c2c9250, 0x1, 0x4004e4fbc8, 0x412ff8, 0x4004e13050) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:389 +0x30 net/textproto.(*Reader).readLineSlice(0x4004e4fcf0, 0x4004e13050, 0x4004e4fc48, 0xffff5c2c9250, 0x2c8, 0xffff8576c5e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/textproto/reader.go:57 +0xac net/textproto.(*Reader).ReadLine(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/textproto/reader.go:38 net/http.ReadResponse(0x4004e56180, 0x4004c41300, 0x1000, 0x4004e58000, 0x4000122490) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/response.go:161 +0x8c github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4004e367b0, 0x4004e4ff10, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:132 +0x2ac github.com/couchbase/cbauth/revrpc.BabysitService(0x4004e4ff10, 0x4004e367b0, 0x0, 0x0, 0x0, 0xd8d03c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth/service.RegisterManager(0x13007d8, 0x4004dc2700, 0x0, 0x0, 0x4004e8df80, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:146 +0xa0 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).registerWithServer(0x4004dc2700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:301 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:205 +0x78 goroutine 195 [select]: net/http.(*persistConn).roundTrip(0x4004e54000, 0x40047cccc0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004c41000, 0xf8, 0x10a8660, 0x4004e8d798) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004c41000, 0x1b6e900, 0xc03eba341d4afbbc, 0x1176f2f3f18) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004c40f00, 0x12db048, 0x1b6e900, 0xc03eba341d4afbbc, 0x1176f2f3f18, 0x1be4a80, 0x0, 0x4000122470, 0x4004e36480, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004e36450, 0x4004c40f00, 0xc03eba341d4afbbc, 0x1176f2f3f18, 0x1be4a80, 0x4000122470, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004e36450, 0x4004c40f00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004e34180, 0x1f, 0x4004e8dd10, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/indexer.getWithAuth(0x4004e34180, 0x1f, 0xe, 0x113846e, 0x11) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:3077 +0x68 github.com/couchbase/indexing/secondary/indexer.getGlobalTopology(0x4004dbf400, 0xe, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:2266 +0x78 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).updateNodeList(0x4004dc2700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:220 +0x3c created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:206 +0x98 goroutine 199 [IO wait]: internal/poll.runtime_pollWait(0xffff5c4484b0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011da18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011da00, 0x4004e5b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011da00, 0x4004e5b000, 0x1000, 0x1000, 0x400242e3e8, 0x7ff1a8, 0x400242e4b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001224a0, 0x4004e5b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004e54000, 0x4004e5b000, 0x1000, 0x1000, 0x60, 0x0, 0x4004e34101) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004e562a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004e562a0, 0x1, 0x4004c71da0, 0x43d584, 0x0, 0xd8c91c, 0x400242e568) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004e54000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 198 [select]: net/http.(*persistConn).roundTrip(0x4004e54240, 0x40047ccf80, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004c41600, 0xf8, 0x10a8660, 0x4004e52588) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004c41600, 0x1b6e900, 0xc03eb9261d507ed0, 0x1bfa30922c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004c41500, 0x12db048, 0x1b6e900, 0xc03eb9261d507ed0, 0x1bfa30922c, 0x1be4a80, 0x0, 0x40001224b0, 0x4004e36cc0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004e36c90, 0x4004c41500, 0xc03eb9261d507ed0, 0x1bfa30922c, 0x1be4a80, 0x40001224b0, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004e36c90, 0x4004c41500, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x40047a1d40, 0x1b, 0x4004e52cf0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004e13290, 0x112f905, 0x6, 0x12db908, 0x40014478c0, 0xee40c0, 0x400011dd18, 0x4004e52cf0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x400003e600, 0x46, 0x12db908, 0x40014478c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x400003e600, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x4004e36bd0, 0xffff5c2c94f8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004e694b0, 0x1063520, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x4004e3e480, 0x4004e3e480, 0x400003e600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.FetchNewClusterInfoCache(0xffffe7eb094e, 0xe, 0x113075a, 0x7, 0x1130e44, 0x8, 0x4004e590c4, 0x2, 0xf01) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:123 +0x134 github.com/couchbase/indexing/secondary/manager.(*IndexManager).FetchNewClusterInfoCache(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:306 github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).getIndexMetadata(0x1be5520, 0x12ec7a0, 0x40047ccf40, 0x40001a3540, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:1284 +0x74 github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).handleIndexMetadataRequest(0x1be5520, 0x12edbe0, 0x4002400ee0, 0x4004c41400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:1271 +0x3e8 net/http.HandlerFunc.ServeHTTP(0x40045fc7b0, 0x12edbe0, 0x4002400ee0, 0x4004c41400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2069 +0x40 net/http.(*ServeMux).ServeHTTP(0x4004502c80, 0x12edbe0, 0x4002400ee0, 0x4004c41400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2448 +0x190 net/http.serverHandler.ServeHTTP(0x400450e540, 0x12edbe0, 0x4002400ee0, 0x4004c41400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2887 +0xbc net/http.(*conn).serve(0x40046ad400, 0x12f86a0, 0x40047ccd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:1952 +0x71c created by net/http.(*Server).Serve /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:3013 +0x318 goroutine 200 [select]: net/http.(*persistConn).writeLoop(0x4004e54000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 204 [IO wait]: internal/poll.runtime_pollWait(0xffff5c4481f8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011de18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011de00, 0x4004e5d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011de00, 0x4004e5d000, 0x1000, 0x1000, 0x400242e3e8, 0x7ff1a8, 0x400242e4b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001224c0, 0x4004e5d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004e54240, 0x4004e5d000, 0x1000, 0x1000, 0x60, 0x0, 0x4004e34101) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004e567e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004e567e0, 0x1, 0x4004e6a1e0, 0x43d584, 0x0, 0xd8c91c, 0x400242e568) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004e54240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 205 [select]: net/http.(*persistConn).writeLoop(0x4004e54240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc Initializing write barrier = 8000 2021-08-16T13:48:21.662+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:48:21.662+00:00 [Info] Setting ipv6=false 2021-08-16T13:48:21.667+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:48:21.669+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:48:21.669+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:48:21.669+00:00 [Info] Setting log level to Info 2021-08-16T13:48:21.669+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:48:21.673+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:48:21 audit: created new audit service 2021-08-16T13:48:21.679+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:48:21.679+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:48:21.680+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:48:21.680+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:48:21.680+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:48:21.690+00:00 [Info] security context initialized 2021-08-16T13:48:21.690+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:48:21.690+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:48:21.690+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:48:21.690+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:48:21.690+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:48:21.702+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:48:21.703+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:48:21.711+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:48:21.727+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:48:21.728+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:48:21.729+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:48:21.729+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:48:21.731+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:48:21.731+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:48:21.733+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:48:21.735+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:48:21.779+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:48:21.780+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 347 us 2021-08-16T13:48:21.780+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:48:21.782+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:48:21.782+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:48:21.786+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:48:21.786+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:48:21.786+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:48:21.786+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:48:21.786+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:48:21.786+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:21.786+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:21.789+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:48:21.789+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:48:21.789+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:48:21.791+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:21.791+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:48:21.791+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:48:21.791+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:48:21.791+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:48:21.792+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:48:21.792+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:48:21.792+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:48:21.795+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 3.687751ms 2021-08-16T13:48:21.796+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:48:21.797+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:48:21.797+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:21.797+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:48:21.797+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:48:21.797+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:48:21.797+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:48:21.807+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:48:21.807+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:48:21.808+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:48:21.808+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:48:21.808+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x4004d2849f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x4004d2849f pc=0xbd8ea4] goroutine 185 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004cf49f0 sp=0x4004cf49c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004cf4a30 sp=0x4004cf49f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x4004d184e0, 0x4004b71800, 0x11a9890, 0x4004cf4b40, 0x4004d184f8, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004cf4ab0 sp=0x4004cf4a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x4004d184e0, 0x4004b71800, 0x11a9890, 0x4004cf4b40, 0x4004d184f8, 0xffff9f212f70, 0x4004cf4c18, 0x49) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004cf4af0 sp=0x4004cf4ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4000032dc0, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004cf4da0 sp=0x4004cf4af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4000032dc0, 0x4000211a40, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004cf4e60 sp=0x4004cf4da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004d16b50, 0x4004d16b60, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004cf5200 sp=0x4004cf4e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004d16b50, 0x4004d16b60, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004cf58f0 sp=0x4004cf5200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004d7b130, 0x4004d07180, 0x40001ea100, 0x4004d16b30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004cf5fb0 sp=0x4004cf58f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004cf5fb0 sp=0x4004cf5fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x4001432000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x400021aea0, 0x4, 0x1136a87, 0xf, 0x4000095e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 20 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x40001aa630) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 21 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x40001aa648) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 22 [IO wait]: internal/poll.runtime_pollWait(0xffff9f216e88, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011c198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011c180, 0x40000a5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011c180, 0x40000a5000, 0x1000, 0x1000, 0x1311860, 0x47b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010030, 0x40000a5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x4000082300, 0x40000b6001, 0xdff, 0xdff, 0x40001998e8, 0x6902d8, 0x4000091948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400000e048, 0x40000b6001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40000a2160, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40000a2160, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40000a2160, 0xf5a960, 0x40000823e0, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x40000823c0, 0x4000046160, 0x4000091b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x40000ae000, 0x12f8f98, 0x40000823c0, 0x40000368d7, 0x16, 0x400000c480, 0x4000199c01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x40000ae000, 0x12f8f98, 0x40000823c0, 0x40000ae000, 0x40000360e8, 0x40000360f0, 0x40000b0000, 0x4000046080, 0xee0980, 0x400000c480, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x40000ae000, 0x12f8f98, 0x40000823c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4000195e00, 0x4000199f78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400019df78, 0x4000195e00, 0x12db8a8, 0x4000126030, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x4000195e00, 0x40001bd4a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x4000195e00, 0x40001bd4a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 6 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400000a600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 23 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 24 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 25 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 26 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 27 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 28 [syscall]: syscall.Syscall(0x3f, 0x0, 0x4000404fa0, 0x4, 0x4000404e68, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x4000404fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x40001ba000, 0x4000404fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x40001b8000, 0x4000404fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 11 [IO wait]: internal/poll.runtime_pollWait(0xffff9f216da0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011c218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011c200, 0x40000b7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011c200, 0x40000b7000, 0x1000, 0x1000, 0x400006ba88, 0x68f7a4, 0x40000a2cb0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4001436020, 0x40000b7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c360, 0x40000b7000, 0x1000, 0x1000, 0xf12180, 0x400012a1d0, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400142c420) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400142c420, 0xa, 0x1008880, 0x400012a190, 0x199, 0xad9, 0x400006bb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400142c420, 0x2, 0x0, 0x9, 0x56b, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4001430c90) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4001430c90, 0x40000ec000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400143c280, 0x40000ec000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400143c280, 0x40000ec000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400143c2c0, 0x40000ec000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40000a2c60, 0x0, 0x400006beb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40000a2c60, 0x0, 0x0, 0x400006bf70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40000a2c60, 0xee0d40, 0x400012a190, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4000096630, 0x400142c8a0, 0x400142ed80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 9 [select]: net/http.(*persistConn).readLoop(0x400000c360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 10 [select]: net/http.(*persistConn).writeLoop(0x400000c360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 50 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400022e2b0, 0x12c18bc, 0x1, 0x400019be58, 0x400142e6c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x4000401ec8, 0x400142e6c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400019bfa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x400143c000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 51 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x400143c000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 60 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000116640) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 54 [IO wait]: internal/poll.runtime_pollWait(0xffff9f216cb8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000b0198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000b0180, 0x400014c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000b0180, 0x400014c000, 0x1000, 0x1000, 0x4000154be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011a058, 0x400014c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003ea6c0, 0x400014c000, 0x1000, 0x1000, 0x4000154cf8, 0x67867c, 0x40000383c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400011e4e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x400011e4e0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40003ea6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 55 [select]: net/http.(*persistConn).writeLoop(0x40003ea6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 58 [select]: net/http.(*persistConn).readLoop(0x40003eaa20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 59 [select]: net/http.(*persistConn).writeLoop(0x40003eaa20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 30 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000081380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 62 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x40002008c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 63 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x40002008c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 64 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x40002008c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 65 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x40002008c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 112 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000080580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 32 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x40004488a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 31 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x400011e2a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 16 [IO wait]: internal/poll.runtime_pollWait(0xffff9f2163a8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000208698, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000208680, 0x40044fb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000208680, 0x40044fb000, 0x1000, 0x1000, 0x40000cd688, 0x69f178, 0x40000cd6f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000101b8, 0x40044fb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c65a0, 0x40044fb000, 0x1000, 0x1000, 0xffff77ef5c00, 0x100000000000, 0xffff780eefff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000083260) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000083260, 0x40a, 0x40000d8000, 0x0, 0xffff00001000, 0x400, 0x400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000083260, 0xffff9f237720, 0x1c15a70, 0x18, 0xffff9f237720, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400010ef30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400010ef30, 0x40044ff000, 0x1000, 0x1000, 0x40000cd998, 0x44e884, 0x6) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000116680, 0x40044ff000, 0x1000, 0x1000, 0x40000834f8, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000116680, 0x40044ff000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001166c0, 0x40044ff000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40000cdd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000cdd80, 0xa, 0x40044ff201, 0x1, 0xffff5ffc2c20, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40000cdd80, 0x40044ff20a, 0x1, 0x40023ed860, 0x0, 0x40044ff259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40000cdd80, 0x400247fa0a, 0x40023ed860, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400143ac70, 0x400247cf00, 0x1f, 0x40000cdee0, 0x400247a948, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x400143ac70, 0x113075a, 0x7, 0x4002406a30, 0xd, 0x400247a948, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x400143ac30, 0x4002406a30, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 14 [select]: net/http.(*persistConn).readLoop(0x400000c900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 15 [select]: net/http.(*persistConn).writeLoop(0x400000c900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 37 [IO wait]: internal/poll.runtime_pollWait(0xffff9f216578, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011c398, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011c380, 0x40000ed000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011c380, 0x40000ed000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40014360b8, 0x40000ed000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c900, 0x40000ed000, 0x1000, 0x1000, 0x4001417200, 0x200000003, 0x4001417200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400142cd20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400142cd20, 0x40000f600a, 0x40001458c8, 0x4d3b10, 0x40001ba060, 0x40000f6000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400142cd20, 0x0, 0x1, 0x0, 0x40001ba078, 0x40000f6000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40023ee630) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40023ee630, 0x4000016000, 0x1000, 0x1000, 0x40001459b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000080300, 0x4000016000, 0x1000, 0x1000, 0x4000082f58, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000080300, 0x4000016000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000080340, 0x4000016000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000145dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000145dd0, 0xa, 0x4000016201, 0x1, 0xd1f, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000145dd0, 0x400001620a, 0x1, 0x4002406a19, 0x0, 0x40000162e1, 0x1, 0xd1f, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000145dd0, 0x40001fd00a, 0x4002406a19, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400143ac70, 0x40001224c8, 0x17, 0x4000145f20, 0x400017d1a0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x400143ac70, 0x113075a, 0x7, 0x400017d1a0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x400143ac30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 38 [IO wait]: internal/poll.runtime_pollWait(0xffff9f216748, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000b0c98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000b0c80, 0x4000017000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000b0c80, 0x4000017000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011a100, 0x4000017000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003eaa20, 0x4000017000, 0x1000, 0x1000, 0x4001417380, 0x200000003, 0x4001417380) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400011f0e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400011f0e0, 0x40000f600a, 0x400014b8c8, 0x4d3b10, 0x40001ba060, 0x40000f6000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400011f0e0, 0x0, 0x1, 0x0, 0x40001ba078, 0x40000f6000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40023ee690) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40023ee690, 0x4000091000, 0x1000, 0x1000, 0x400014b9b8, 0x0, 0x400014b988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000080380, 0x4000091000, 0x1000, 0x1000, 0x4000102780, 0x3, 0x400014bbe0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000080380, 0x4000091000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40000803c0, 0x4000091000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400014bdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400014bdd0, 0xa, 0x4000091201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400014bdd0, 0x400009120a, 0x1, 0x40024fefbb, 0x0, 0x4000091263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400014bdd0, 0x400004620a, 0x40024fefbb, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400143ac70, 0x40000423f0, 0x24, 0x400014bf20, 0x400017d188, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x400143ac70, 0x113075a, 0x7, 0x400017d188, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x400143ac30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 48 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x400143d7c0, 0x40000decb0, 0x40000ded20, 0x40047289d0, 0x12e8f90, 0x4000200700, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x4002575fc4, 0x5, 0x400143d680, 0x12eca70, 0x400143d500, 0x130d478, 0x40001f81c0, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x4000200b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x4000200b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 153 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4004d1e180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 46 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400143d100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 47 [IO wait]: internal/poll.runtime_pollWait(0xffff9f216490, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004408b18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4004408b00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4004408b00, 0x58ec48, 0x400142fd40, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x400476d4e8, 0x4714f4, 0x400142fd40, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x400476d4e8, 0x4000111f00, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x400143d680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 45 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40003ecf80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 84 [select]: net/http.(*persistConn).readLoop(0x40001c65a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 85 [select]: net/http.(*persistConn).writeLoop(0x40001c65a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 33 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x40000b0080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 98 [IO wait]: internal/poll.runtime_pollWait(0xffff9f2162c0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000b0218, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000b0200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000b0200, 0x0, 0x400453ae98, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4000126300, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4000126300, 0x12ed9d0, 0x4000126300, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x40000b0100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 99 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4000468a90, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 100 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4000468a90, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 101 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x4000468a90) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 102 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x4000449380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 78 [IO wait]: internal/poll.runtime_pollWait(0xffff9f2160f0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000208d18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000208d00, 0x40045b5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000208d00, 0x40045b5000, 0x1000, 0x1000, 0x4000151a48, 0x652ce0, 0x12edac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b8530, 0x40045b5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c6c60, 0x40045b5000, 0x1000, 0x1000, 0x113674e, 0x412bc0, 0x400046e0f0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400011eea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400011eea0, 0xa, 0x0, 0x0, 0x0, 0x0, 0x4004541c38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400011eea0, 0x0, 0x4004541c78, 0x655784, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004bc46c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004bc46c0, 0x4004b9d200, 0x200, 0x200, 0x4, 0x0, 0x4004541d08) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40001d9940, 0x4004b9d200, 0x200, 0x200, 0x40001204b7, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40001d9940, 0x4004b9d200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001d9980, 0x4004b9d200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004bc1760, 0x0, 0x4004541eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004bc1760, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004bc1760, 0xee0d40, 0x40000aedc0, 0x12db068, 0x40001d9980) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004b610e0, 0x4004549c80, 0x400467d080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 41 [select]: net/http.(*persistConn).readLoop(0x4000464480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 105 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x400027e770) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 49 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x40002275e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 152 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x4004452a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 42 [select]: net/http.(*persistConn).writeLoop(0x4000464480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 108 [select]: net/http.(*persistConn).readLoop(0x40001c6c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 109 [select]: net/http.(*persistConn).writeLoop(0x40001c6c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 114 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x40002275e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 115 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x400143d7c0, 0x40047289d0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 116 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x4000200700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 117 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x4000200700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 118 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x40024d2fc0, 0x4004772300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 119 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x4000080540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 120 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x4000227680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 121 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x4000227680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 122 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x40014363b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 123 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x4000227680, 0x4004772360) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 124 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400022e2b0, 0x1144559, 0x1b, 0x40000c9e38, 0x4004772480, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x40047294f0, 0x4004772480, 0xf1f700, 0x400022fb30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40000c9f80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x400253ba40, 0x40047294f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 125 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 126 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400012ad20, 0xd00007, 0x4004774fb8, 0x5, 0x4004729500, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 127 [IO wait]: internal/poll.runtime_pollWait(0xffff9f216660, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004408d98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4004408d80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4004408d80, 0xffff9f292f18, 0x30, 0x4004cb4540) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x400476d920, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x400476d920, 0x4004cb4540, 0x11ae500, 0x4000200cc0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x4000200c40, 0x12ed9d0, 0x400476d920, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x4000200c40, 0x12ed9d0, 0x400476d920, 0x4001432000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 128 [IO wait]: internal/poll.runtime_pollWait(0xffff9f216008, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004408e98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4004408e80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4004408e80, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x400476d938, 0xffff9f292f18, 0xffff74bdd0e8, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x400476d938, 0x665e1c, 0x0, 0xfe5e60, 0x40045b65c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x400476d950, 0x4004cb45d0, 0x11ae500, 0x4000200da0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x4000200d20, 0x12ec710, 0x400476d950, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x4000200d20, 0x12ec710, 0x400476d950, 0x4001432000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 92 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40000bd290, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 150 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x4004452a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 80 [IO wait]: internal/poll.runtime_pollWait(0xffff5f34b640, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011d518, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011d500, 0x4004be5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011d500, 0x4004be5000, 0x1000, 0x1000, 0x400012cbe0, 0x469b10, 0x400477f500) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011a460, 0x4004be5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003eb560, 0x4004be5000, 0x1000, 0x1000, 0x400012ccf8, 0x67867c, 0x400467d260) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004549ce0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004549ce0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40003eb560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 81 [select]: net/http.(*persistConn).writeLoop(0x40003eb560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 151 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x4004452a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 149 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x4004452a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 148 [semacquire]: sync.runtime_Semacquire(0x4004d7b138) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004d7b130) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x40001ea100, 0x400008c600, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffcd899969, 0x29, 0xffffcd8999cb, 0x25, 0x4004d268a0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x40046f52a0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffcd899969, 0x29, 0xffffcd8999cb, 0x25, 0x4004d268a0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x40046f52a0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004cf5190, 0x4004cf4e60, 0x400008c600, 0x4002404000, 0x4000116640, 0x0, 0xbf, 0x40000c6da8, 0xffff74be4f28, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x4001432000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x40046f52a0, 0xb, 0x40046f5298, 0x6, 0x40046f52b0, 0xd, 0x40046e9360, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x4001432000, 0x40003b0000, 0x4001430030) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x4001432000, 0x40004568c0, 0x2, 0x2, 0x40004568d0, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 163 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x4004452a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 93 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40000bd290, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 94 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x40000bd290) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 164 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400022e2b0, 0x1144559, 0x1b, 0x40000d3e38, 0x400467d8c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x40046c15a0, 0x400467d8c0, 0x40045b9eb8, 0x6a1ed4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1008880, 0x4001431c01) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40000d3f80, 0x12db068, 0x4004e0c000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004549f20, 0x40046c15a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 158 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4004d1e1c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 156 [IO wait]: internal/poll.runtime_pollWait(0xffff5f34b728, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011d698, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011d680, 0x4004d0b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011d680, 0x4004d0b000, 0x1000, 0x1000, 0x40024d7be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011a4c0, 0x4004d0b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004d08120, 0x4004d0b000, 0x1000, 0x1000, 0x40024d7cf8, 0x67867c, 0x4004d26000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004d22240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004d22240, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004d08120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 157 [select]: net/http.(*persistConn).writeLoop(0x4004d08120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 134 [IO wait]: internal/poll.runtime_pollWait(0xffff9f2161d8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000b0418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000b0400, 0x400250e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000b0400, 0x400250e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40014360f0, 0x400250e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000464480, 0x400250e000, 0x1000, 0x1000, 0x0, 0x0, 0x400012fb28) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400142dc20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400142dc20, 0xa, 0x40004073e8, 0x48731c, 0x9, 0x1, 0x4000407489) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400142dc20, 0x0, 0x4000407448, 0x57d438, 0x11320c7, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004cb5110) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004cb5110, 0x4004cf6400, 0x200, 0x200, 0x4004775038, 0x4004775038, 0x4000120c30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400143de80, 0x4004cf6400, 0x200, 0x200, 0x877b4b68a8, 0x2ee4f173, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400143de80, 0x4004cf6400, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400143dec0, 0x4004cf6400, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004daa420, 0x0, 0x40004076b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004daa420, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004daa420, 0xee0d40, 0x400012b3b0, 0x12db068, 0x400143dec0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004721c20, 0x4004cc0840, 0x4004772cc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 159 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x400462b080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 160 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x400462b080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 161 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x400462b080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 178 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x400462b080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 179 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004d1aa00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 180 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x4004d1aa00, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 181 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x4001432000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 182 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x4001432000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 183 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x4001432000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 184 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:48:26.952+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:48:26.952+00:00 [Info] Setting ipv6=false 2021-08-16T13:48:26.957+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:48:26.959+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:48:26.959+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:48:26.959+00:00 [Info] Setting log level to Info 2021-08-16T13:48:26.959+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:48:26.961+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:48:26 audit: created new audit service 2021-08-16T13:48:26.969+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:48:26.969+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:48:26.970+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:48:26.970+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:48:26.970+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:48:26.980+00:00 [Info] security context initialized 2021-08-16T13:48:26.980+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:48:26.980+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:48:26.980+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:48:26.980+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:48:26.980+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:48:26.993+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:48:27.002+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:48:27.009+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:48:27.023+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:48:27.023+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:48:27.024+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:48:27.025+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:48:27.026+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:48:27.027+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:48:27.027+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:48:27.033+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:48:27.077+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:48:27.078+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 56 us 2021-08-16T13:48:27.078+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:48:27.080+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:48:27.081+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:48:27.081+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:48:27.081+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:48:27.081+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:48:27.081+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:48:27.081+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:48:27.084+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:27.084+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:27.086+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:48:27.086+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:48:27.086+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:48:27.086+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:27.087+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:48:27.087+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:48:27.087+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:48:27.087+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:48:27.087+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:48:27.087+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:48:27.087+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:48:27.091+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 4.298372ms 2021-08-16T13:48:27.091+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:48:27.093+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:48:27.093+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:27.093+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:48:27.093+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:48:27.093+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:48:27.093+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... unexpected fault address 0x400014fb1f fatal error: fault 2021-08-16T13:48:27.103+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:48:27.103+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:48:27.103+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:48:27.103+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:48:27.103+00:00 [Info] : Recover shard shard1 from metadata completed. [signal SIGBUS: bus error code=0x1 addr=0x400014fb1f pc=0xbd8ea4] goroutine 184 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004bea9f0 sp=0x4004bea9c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004beaa30 sp=0x4004bea9f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x4004ede000, 0x4004ee4000, 0x11a9890, 0x4004beab40, 0x4004ede018, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004beaab0 sp=0x4004beaa40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x4004ede000, 0x4004ee4000, 0x11a9890, 0x4004beab40, 0x4004ede018, 0xffff86ab9758, 0x4004beac18, 0x49) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004beaaf0 sp=0x4004beaab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x400015edc0, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004beada0 sp=0x4004beaaf0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x400015edc0, 0x40013e16c0, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004beae60 sp=0x4004beada0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x40046affe0, 0x40046afff0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004beb200 sp=0x4004beae60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x40046affe0, 0x40046afff0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004beb8f0 sp=0x4004beb200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004eca950, 0x4004ebae00, 0x4004615080, 0x40046affc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004bebfb0 sp=0x4004beb8f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004bebfb0 sp=0x4004bebfb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x4000168380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x4000181650, 0x4, 0x1136a87, 0xf, 0x400011de68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 18 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x40001a80d8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 19 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x40001a80f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 20 [IO wait]: internal/poll.runtime_pollWait(0xffff5c378108, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000132098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000132080, 0x4000136000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000132080, 0x4000136000, 0x1000, 0x1000, 0x1311860, 0x47b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400018c090, 0x4000136000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x400010e2a0, 0x4000137001, 0xdff, 0xdff, 0x400009f8e8, 0x6902d8, 0x4000119948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400011e030, 0x4000137001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x4000146000, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4000146000, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4000146000, 0xf5a960, 0x400010e380, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x400010e360, 0x4000124160, 0x4000119b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x4000140000, 0x12f8f98, 0x400010e360, 0x4000122897, 0x16, 0x400014c120, 0x400009fc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x4000140000, 0x12f8f98, 0x400010e360, 0x4000140000, 0x40001220a8, 0x40001220b0, 0x4000132100, 0x4000124080, 0xee0980, 0x400014c120, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x4000140000, 0x12f8f98, 0x400010e360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x40001804b0, 0x400009ff78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x4001405f78, 0x40001804b0, 0x12db8a8, 0x400011e018, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x40001804b0, 0x400019e5a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x40001804b0, 0x400019e5a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 21 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 22 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 23 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 24 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 25 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 7 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400000a700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 26 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40003f67a0, 0x4, 0x40003f6668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40003f67a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x40003f67a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x40003f67a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 12 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000184770, 0x12c18bc, 0x1, 0x4001407e58, 0x40001b6180, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x40003f8ec8, 0x40001b6180, 0x0, 0x40001b6538) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x4714f4, 0x40001b64e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4001407fa0, 0x4000122000, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4000128000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 10 [select]: net/http.(*persistConn).readLoop(0x400014c120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 11 [select]: net/http.(*persistConn).writeLoop(0x400014c120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 13 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4000128000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 74 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x40013de680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 16 [IO wait]: internal/poll.runtime_pollWait(0xffff5c377f38, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000132398, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000132380, 0x40000ce000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000132380, 0x40000ce000, 0x1000, 0x1000, 0x4000064be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000120120, 0x40000ce000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400014c7e0, 0x40000ce000, 0x1000, 0x1000, 0x4000064cf8, 0x67867c, 0x40001b69c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400010eb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x400010eb40, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400014c7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 66 [select]: net/http.(*persistConn).writeLoop(0x400014c7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 35 [IO wait]: internal/poll.runtime_pollWait(0xffff5c378020, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000132298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000132280, 0x40000fe000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000132280, 0x40000fe000, 0x1000, 0x1000, 0x4000118a88, 0x68f7a4, 0x400016e050) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000120080, 0x40000fe000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400014c120, 0x40000fe000, 0x1000, 0x1000, 0xf12180, 0x40001941d0, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400010e6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400010e6c0, 0xa, 0x1008880, 0x4000194190, 0x199, 0x1000, 0x4000118b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400010e6c0, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000096d50) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000096d50, 0x4000172000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000080000, 0x4000172000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000080000, 0x4000172000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000080140, 0x4000172000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x400016e000, 0x0, 0x4000118eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400016e000, 0x0, 0x0, 0x4000118f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400016e000, 0xee0d40, 0x4000194190, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400016a000, 0x40000827e0, 0x40013ae660) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 40 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000188d80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 38 [select]: net/http.(*persistConn).readLoop(0x400000cc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 39 [select]: net/http.(*persistConn).writeLoop(0x400000cc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 71 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40013e4000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 73 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x4002500de0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 72 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x40000832c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 53 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x40000e87e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 125 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000189440) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 28 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x40000e87e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 29 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x40000e87e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 30 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x40000e87e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 46 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4000099a00, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 119 [IO wait]: internal/poll.runtime_pollWait(0xffff5c3779c8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400018ef98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400018ef80, 0x4002372000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400018ef80, 0x4002372000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010578, 0x4002372000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000cc60, 0x4002372000, 0x1000, 0x1000, 0x20, 0x30, 0x20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40000834a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000834a0, 0x100a, 0x0, 0x0, 0x0, 0x0, 0x40003f7c38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40000834a0, 0x7356e4, 0xffffe892b94e, 0xe, 0x40003f7ca8, 0x40b2f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004687c80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004687c80, 0x4004ccc200, 0x200, 0x200, 0xffffe892b94e, 0xe, 0x4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400238be00, 0x4004ccc200, 0x200, 0x200, 0x4000182ba7, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400238be00, 0x4004ccc200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400238be40, 0x4004ccc200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004cc6dc0, 0x680d00, 0x40003f7eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004cc6dc0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004cc6dc0, 0xee0d40, 0x4000194e10, 0x12db068, 0x400238be40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40047c8fc0, 0x4004782de0, 0x4004cc4720) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 109 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x4000081bc0, 0x400016d420, 0x400016d490, 0x40047536f0, 0x12e8f90, 0x40000e8460, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x4004c0f3a4, 0x5, 0x4000081740, 0x12eca70, 0x40013e5f80, 0x130d478, 0x400019c0e0, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40003acee0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40003acee0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 32 [select]: net/http.(*persistConn).readLoop(0x4004410000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 33 [select]: net/http.(*persistConn).writeLoop(0x4004410000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 75 [IO wait]: internal/poll.runtime_pollWait(0xffff5c3778e0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40013de798, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40013de780, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40013de780, 0x0, 0x40024b8e98, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40023ed428, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40023ed428, 0x12ed9d0, 0x40023ed428, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x40013de700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 76 [IO wait]: internal/poll.runtime_pollWait(0xffff5c3777f8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40013de898, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40013de880, 0x4002430000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40013de880, 0x4002430000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40013fa0e8, 0x4002430000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40023fa6c0, 0x4002430000, 0x1000, 0x1000, 0x4001398c00, 0x200000003, 0x4001398c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000083bc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000083bc0, 0x400014800a, 0x400242d8c8, 0x4d3b10, 0x40000820c0, 0x4000148000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000083bc0, 0x0, 0x1, 0x0, 0x40000820d8, 0x4000148000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40001116e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40001116e0, 0x400447e000, 0x1000, 0x1000, 0x400242d9b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000128480, 0x400447e000, 0x1000, 0x1000, 0x40000837f8, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000128480, 0x400447e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001284c0, 0x400447e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400242ddd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400242ddd0, 0xa, 0x400447e201, 0x1, 0xd1f, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400242ddd0, 0x400447e20a, 0x1, 0x4000123a59, 0x0, 0x400447e2e1, 0x1, 0xd1f, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400242ddd0, 0x4000144f0a, 0x4000123a59, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000144ee0, 0x400003adc8, 0x17, 0x400242df20, 0x40023ed488, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x4000144ee0, 0x113075a, 0x7, 0x40023ed488, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x4000144ea0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 77 [IO wait]: internal/poll.runtime_pollWait(0xffff5c377710, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000132598, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000132580, 0x40024bc000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000132580, 0x40024bc000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010008, 0x40024bc000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004410000, 0x40024bc000, 0x1000, 0x1000, 0x4001398d80, 0x200000003, 0x4001398d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40013ac420) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40013ac420, 0x400014800a, 0x40000a18c8, 0x4d3b10, 0x40000820c0, 0x4000148000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40013ac420, 0x0, 0x1, 0x0, 0x40000820d8, 0x4000148000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4002501ce0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4002501ce0, 0x4002436000, 0x1000, 0x1000, 0x40000a19b8, 0x0, 0x40000a1988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40013e4940, 0x4002436000, 0x1000, 0x1000, 0x4000001e00, 0x3, 0x40000a1be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40013e4940, 0x4002436000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40013e4980, 0x4002436000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40000a1dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000a1dd0, 0xa, 0x4002436201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40000a1dd0, 0x400243620a, 0x1, 0x4000036bcb, 0x0, 0x4002436263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40000a1dd0, 0x400018a80a, 0x4000036bcb, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000144ee0, 0x4000042900, 0x24, 0x40000a1f20, 0x40023ed470, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x4000144ee0, 0x113075a, 0x7, 0x40023ed470, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x4000144ea0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 98 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x400019c070) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 80 [select]: net/http.(*persistConn).readLoop(0x40023fa6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 81 [select]: net/http.(*persistConn).writeLoop(0x40023fa6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 47 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4000099a00, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 48 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x4000099a00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 49 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x40024ae810) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 107 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000128e40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 108 [IO wait]: internal/poll.runtime_pollWait(0xffff5c377540, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40013dfb18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40013dfb00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40013dfb00, 0x58ec48, 0x4000039aa0, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c2eac8, 0x4714f4, 0x4000039aa0, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c2eac8, 0x4002471600, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x4000081740) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 118 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000188e80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 55 [IO wait]: internal/poll.runtime_pollWait(0xffff5c377458, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000132498, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000132480, 0x400251f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000132480, 0x400251f000, 0x1000, 0x1000, 0x4002429688, 0x69f178, 0x40024296f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400018c190, 0x400251f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400014cea0, 0x400251f000, 0x1000, 0x1000, 0x5, 0x400252e000, 0x40024297c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400037d380) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400037d380, 0x68f90a, 0x40001f9290, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400037d380, 0x0, 0x4002429868, 0x41a0a8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40024aec60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40024aec60, 0x4002523000, 0x1000, 0x1000, 0x4002429998, 0x44e884, 0x1c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000188640, 0x4002523000, 0x1000, 0x1000, 0x40000837f8, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000188640, 0x4002523000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000188680, 0x4002523000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4002429d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002429d80, 0xa, 0x4002523201, 0x1, 0xffff86abb380, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4002429d80, 0x400252320a, 0x1, 0x4002381a60, 0x0, 0x4002523259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4002429d80, 0x400016e50a, 0x4002381a60, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000144ee0, 0x4004475740, 0x1f, 0x4002429ee0, 0x40024ab4e8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x4000144ee0, 0x113075a, 0x7, 0x4000123a70, 0xd, 0x40024ab4e8, 0x0, 0x4000184c10, 0x40001856c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x4000144ea0, 0x4000123a70, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 91 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x400253c0e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 84 [select]: net/http.(*persistConn).readLoop(0x400014cea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 89 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x400253c0e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 88 [semacquire]: sync.runtime_Semacquire(0x4004eca958) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004eca950) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x4004615080, 0x4000111110, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffe892b969, 0x29, 0xffffe892b9cb, 0x25, 0x4004cc5680, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004c6a090, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffe892b969, 0x29, 0xffffe892b9cb, 0x25, 0x4004cc5680, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004c6a090, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004beb190, 0x4004beae60, 0x4000111110, 0x40001eb680, 0x4000188d80, 0x0, 0xffff5c37bf78, 0x78, 0xffff86aafa90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x4000168380, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004c6a090, 0xb, 0x4004c6a088, 0x6, 0x4004c6a0a0, 0xd, 0x4004c2cb20, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x4000168380, 0x4000370000, 0x4000110240) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x4000168380, 0x4002388f10, 0x2, 0x2, 0x4002388f30, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 85 [select]: net/http.(*persistConn).writeLoop(0x400014cea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 58 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400238acc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 110 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40000e8460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 111 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40000e8460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 112 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x4002439050, 0x400477cba0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 113 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x4000188e40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 130 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x400017f540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 131 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x400017f540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 132 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x40013fa4e8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 133 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x400017f5e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 134 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x400017f5e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 135 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x4000081bc0, 0x40047536f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 59 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x400017f540, 0x40044f2900) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 60 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000184770, 0x1144559, 0x1b, 0x400009be38, 0x40044f2a20, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x40044fe7a0, 0x40044f2a20, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400009bf80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004bd64e0, 0x40044fe7a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 61 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 62 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x4000140820, 0xd00007, 0x40044b7728, 0x5, 0x40044fe7b0, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 63 [IO wait]: internal/poll.runtime_pollWait(0xffff5c377370, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000132c18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000132c00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000132c00, 0xffff86aaff18, 0x30, 0x4004bd85d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40044dd380, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40044dd380, 0x4004bd85d0, 0x11ae500, 0x40000e85c0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40000e8540, 0x12ed9d0, 0x40044dd380, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40000e8540, 0x12ed9d0, 0x40044dd380, 0x4000168380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 64 [IO wait]: internal/poll.runtime_pollWait(0xffff5c377288, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000132d18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000132d00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000132d00, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40044dd398, 0xffff86aaff18, 0xffff5c3da448, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40044dd398, 0x665e1c, 0x0, 0xfe5e60, 0x40024b5dc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x40044dd3b0, 0x4004bd8660, 0x11ae500, 0x40000e86a0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40000e8620, 0x12ec710, 0x40044dd3b0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40000e8620, 0x12ec710, 0x40044dd3b0, 0x4000168380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 151 [select]: net/http.(*persistConn).writeLoop(0x4004530000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 90 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x400253c0e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 121 [IO wait]: internal/poll.runtime_pollWait(0xffff5c0eabc0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004cd4098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004cd4080, 0x4004ccf000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004cd4080, 0x4004ccf000, 0x1000, 0x1000, 0x40013b7be0, 0x469b10, 0x4004c78900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010898, 0x4004ccf000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000cfc0, 0x4004ccf000, 0x1000, 0x1000, 0x40013b7cf8, 0x67867c, 0x4004cc4900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004782e40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004782e40, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000cfc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 122 [select]: net/http.(*persistConn).writeLoop(0x400000cfc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 148 [IO wait]: internal/poll.runtime_pollWait(0xffff5c0eaca8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000133018, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000133000, 0x4004bff000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000133000, 0x4004bff000, 0x1000, 0x1000, 0x4002463be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001202e0, 0x4004bff000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400014de60, 0x4004bff000, 0x1000, 0x1000, 0x4002463cf8, 0x67867c, 0x40044f35c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004bd70e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004bd70e0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400014de60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 149 [select]: net/http.(*persistConn).writeLoop(0x400014de60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 142 [IO wait]: internal/poll.runtime_pollWait(0xffff5c0eaad8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0998, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0980, 0x400451d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0980, 0x400451d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400018c3a8, 0x400451d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004530120, 0x400451d000, 0x1000, 0x1000, 0x0, 0x0, 0x4002525b28) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400257ef60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400257ef60, 0x4001cc800a, 0x4001ce8000, 0x4001d08000, 0x4001d28000, 0x4001d48000, 0x4001d68000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400257ef60, 0x4001e88000, 0x4001ea8000, 0x4001ec8000, 0x4001ee8000, 0x4001f08000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004db2540) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004db2540, 0x4004c41000, 0x200, 0x200, 0x40021e8000, 0x4002208000, 0x4002228000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004db8180, 0x4004c41000, 0x200, 0x200, 0x400037a360, 0x400037b2c0, 0x4000190ba0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004db8180, 0x4004c41000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004db81c0, 0x4004c41000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004dbe420, 0x4000191400, 0x40025076b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004dbe420, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004dbe420, 0xee0d40, 0x40013fd270, 0x12db068, 0x4004db81c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004c31b90, 0x4004c7a900, 0x400477d5c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 92 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x400253c0e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 93 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4000189400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 150 [IO wait]: internal/poll.runtime_pollWait(0xffff5c377628, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0898, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0880, 0x4004e21000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0880, 0x4004e21000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000120300, 0x4004e21000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004530000, 0x4004e21000, 0x1000, 0x1000, 0x60, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004bd72c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004bd72c0, 0x1, 0x40044f38c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004530000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 96 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40001f9ad0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 97 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40001f9ad0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 162 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x40001f9ad0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 163 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x400253c0e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 164 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000184770, 0x1144559, 0x1b, 0x4004529e38, 0x400451a480, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x40003fb6f0, 0x400451a480, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004529f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x400257e480, 0x40003fb6f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 167 [select]: net/http.(*persistConn).readLoop(0x4004530120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 168 [select]: net/http.(*persistConn).writeLoop(0x4004530120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 126 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x40001f9970) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 127 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x40001f9970) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 128 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x40001f9970) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 129 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x40001f9970) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 178 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004ebc400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 179 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x4004ebc400, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 180 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x4000168380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 181 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x4000168380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 182 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x4000168380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 183 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:48:32.273+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:48:32.273+00:00 [Info] Setting ipv6=false 2021-08-16T13:48:32.281+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:48:32.289+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:48:32.289+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:48:32.289+00:00 [Info] Setting log level to Info 2021-08-16T13:48:32.289+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:48:32.292+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021-08-16T13:48:32.300+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:48:32.300+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:48:32.300+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:48:32.300+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:48:32.300+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021/08/16 13:48:32 audit: created new audit service 2021-08-16T13:48:32.330+00:00 [Info] security context initialized 2021-08-16T13:48:32.330+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:48:32.330+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:48:32.330+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:48:32.330+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:48:32.330+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:48:32.346+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:48:32.346+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:48:32.358+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:48:32.372+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:48:32.372+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:48:32.373+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:48:32.375+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:48:32.378+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:48:32.382+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:48:32.382+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:48:32.386+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:48:32.427+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:48:32.427+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 52 us 2021-08-16T13:48:32.427+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:48:32.429+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:48:32.429+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:48:32.433+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:48:32.433+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:48:32.433+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:48:32.433+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:48:32.433+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:48:32.433+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:32.433+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:32.434+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:48:32.434+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:48:32.434+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:48:32.434+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:32.434+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:48:32.434+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:48:32.434+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:48:32.434+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:48:32.434+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:48:32.434+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:48:32.434+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:48:32.440+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 5.74495ms 2021-08-16T13:48:32.440+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:48:32.441+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:48:32.441+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:32.441+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:48:32.442+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:48:32.442+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:48:32.442+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... unexpected fault address 0x4004f2601f 2021-08-16T13:48:32.451+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:48:32.451+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:48:32.452+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:48:32.452+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:48:32.452+00:00 [Info] : Recover shard shard1 from metadata completed. fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x4004f2601f pc=0xbd8ea4] goroutine 172 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004d669f0 sp=0x4004d669c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004d66a30 sp=0x4004d669f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x4004e36680, 0x4004775200, 0x11a9890, 0x4004d66b40, 0x4004e36698, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004d66ab0 sp=0x4004d66a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x4004e36680, 0x4004775200, 0x11a9890, 0x4004d66b40, 0x4004e36698, 0xc18f8c, 0x4004d66c18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004d66af0 sp=0x4004d66ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x40001aec80, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004d66da0 sp=0x4004d66af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x40001aec80, 0x4000253a40, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004d66e60 sp=0x4004d66da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004cb18f0, 0x4004cb1900, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004d67200 sp=0x4004d66e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004cb18f0, 0x4004cb1900, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004d678f0 sp=0x4004d67200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004e7e270, 0x4004e2dc00, 0x40044f6b00, 0x4004cb18d0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004d67fb0 sp=0x4004d678f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004d67fb0 sp=0x4004d67fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x40001fa700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x400010ff20, 0x4, 0x1136a87, 0xf, 0x4000211e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 34 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x40001260f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 35 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x4000126108) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 36 [IO wait]: internal/poll.runtime_pollWait(0xffffa1415758, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000208118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000208100, 0x4000221000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000208100, 0x4000221000, 0x1000, 0x1000, 0x1311860, 0x47b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b84f8, 0x4000221000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x40001ba4e0, 0x4000382001, 0xdff, 0xdff, 0x400019d8e8, 0x6902d8, 0x400020d948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x40001aa5b8, 0x4000382001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x400012e160, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400012e160, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400012e160, 0xf5a960, 0x4000114800, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x40001147e0, 0x4000118560, 0x400020db28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400012a690, 0x12f8f98, 0x40001147e0, 0x4000110fe7, 0x16, 0x4000384120, 0x400019dc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400012a690, 0x12f8f98, 0x40001147e0, 0x400012a690, 0x40001107f8, 0x4000110800, 0x400011e780, 0x4000118480, 0xee0980, 0x4000384120, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400012a690, 0x12f8f98, 0x40001147e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x400010e690, 0x400019df78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400019ff78, 0x400010e690, 0x12db8a8, 0x40001aa588, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x400010e690, 0x4000132500, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x400010e690, 0x4000132500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 38 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 39 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 40 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 41 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 42 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 20 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x4000190400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 43 [syscall]: syscall.Syscall(0x3f, 0x0, 0x4000153fa0, 0x4, 0x4000153e68, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x4000153fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x40001ba000, 0x4000153fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x40001b8000, 0x4000153fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 27 [IO wait]: internal/poll.runtime_pollWait(0xffffa1415670, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000208098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000208080, 0x4000220000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000208080, 0x4000220000, 0x1000, 0x1000, 0x4000066a88, 0x68f7a4, 0x40001e6470) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000116020, 0x4000220000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c6360, 0x4000220000, 0x1000, 0x1000, 0xf12180, 0x400012a1d0, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003fed20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003fed20, 0xa, 0x1008880, 0x400012a190, 0x199, 0xad9, 0x4000066b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003fed20, 0x2, 0x0, 0x9, 0x56b, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400043eff0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400043eff0, 0x4000258000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40001d9580, 0x4000258000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40001d9580, 0x4000258000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001d95c0, 0x4000258000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40001e6420, 0x0, 0x4000066eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40001e6420, 0x0, 0x0, 0x4000066f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40001e6420, 0xee0d40, 0x400012a190, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40001461b0, 0x40003ff1a0, 0x4000144480) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 25 [select]: net/http.(*persistConn).readLoop(0x40001c6360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 26 [select]: net/http.(*persistConn).writeLoop(0x40001c6360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 6 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000112be0, 0x12c18bc, 0x1, 0x400019de58, 0x4000182840, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x400014cec8, 0x4000182840, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400019dfa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x40001d9700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 7 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x40001d9700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 47 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4004426c00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 45 [IO wait]: internal/poll.runtime_pollWait(0xffffa1415588, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000208298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000208280, 0x4000490000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000208280, 0x4000490000, 0x1000, 0x1000, 0x40004a4be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010008, 0x4000490000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c6b40, 0x4000490000, 0x1000, 0x1000, 0x40004a4cf8, 0x67867c, 0x4000038120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000082120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000082120, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40001c6b40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 46 [select]: net/http.(*persistConn).writeLoop(0x40001c6b40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 30 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400011d1c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 117 [IO wait]: internal/poll.runtime_pollWait(0xffffa1415018, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40004e9018, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40004e9000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40004e9000, 0x58ec48, 0x4000145bc0, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004cae0f0, 0x4714f4, 0x4000145bc0, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004cae0f0, 0x4004726a00, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x400472acc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 116 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400472a240) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 107 [select]: net/http.(*persistConn).writeLoop(0x40024aa6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 12 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x40000ac620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 57 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400011d300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 97 [semacquire]: sync.runtime_Semacquire(0x4004e7e278) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004e7e270) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x40044f6b00, 0x400022a7b0, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffcaa4c969, 0x29, 0xffffcaa4c9cb, 0x25, 0x4004e32d20, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004c9f160, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffcaa4c969, 0x29, 0xffffcaa4c9cb, 0x25, 0x4004e32d20, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004c9f160, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004d67190, 0x4004d66e60, 0x400022a7b0, 0x4002480000, 0x400011d1c0, 0x0, 0xffffa14cc5e0, 0x4004c56da8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x40001fa700, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004c9f160, 0xb, 0x4004c9f158, 0x6, 0x4004c9f170, 0xd, 0x4004c992c0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x40001fa700, 0x40003f0000, 0x4000195e60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x40001fa700, 0x400009e3e0, 0x2, 0x2, 0x400009e3f0, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 15 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x40000ac620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 16 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x40000ac620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 50 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x40000ac620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 118 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x4004426e00, 0x4000252e00, 0x4000252e70, 0x400441b8a0, 0x12e8f90, 0x40002007e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x4004c9e474, 0x5, 0x400472acc0, 0x12eca70, 0x400472ab80, 0x130d478, 0x4000130070, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x4000200b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x4000200b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 110 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x40044628c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 109 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x40044628c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 58 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x400248f200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 59 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x4002483080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 60 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x400011e800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 61 [IO wait]: internal/poll.runtime_pollWait(0xffffa1414e48, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011ee18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400011ee00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400011ee00, 0x4714f4, 0x400014ee98, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40024fa690, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40024fa690, 0x12ed9d0, 0x40024fa690, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x400011ed80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 62 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40001fcf70, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 63 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40001fcf70, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 64 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x40001fcf70) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 65 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x4002483b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 82 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40003f0620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 32 [select]: net/http.(*persistConn).readLoop(0x40024aa240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 33 [select]: net/http.(*persistConn).writeLoop(0x40024aa240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 83 [IO wait]: internal/poll.runtime_pollWait(0xffffa1414c78, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000208418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000208400, 0x4000137000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000208400, 0x4000137000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010128, 0x4000137000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c900, 0x4000137000, 0x1000, 0x1000, 0x40024a2d80, 0x200000003, 0x40024a2d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000082c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000082c00, 0x40004ac00a, 0x400047b8c8, 0x4d3b10, 0x40001ba060, 0x40004ac000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000082c00, 0x0, 0x1, 0x0, 0x40001ba078, 0x40004ac000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400448ec90) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400448ec90, 0x4000403000, 0x1000, 0x1000, 0x400047b9b8, 0x0, 0x400047b988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40004927c0, 0x4000403000, 0x1000, 0x1000, 0x4000187b00, 0x3, 0x400047bbe0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40004927c0, 0x4000403000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000492800, 0x4000403000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400047bdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400047bdd0, 0xa, 0x4000403201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400047bdd0, 0x400040320a, 0x1, 0x40001b4199, 0x0, 0x40004032e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400047bdd0, 0x40004fce0a, 0x40001b4199, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40004fcad0, 0x400003ac30, 0x17, 0x400047bf20, 0x400007f608, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x40004fcad0, 0x113075a, 0x7, 0x400007f608, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x40004fca90) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 84 [IO wait]: internal/poll.runtime_pollWait(0xffffa1414d60, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011ef98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011ef80, 0x4002573000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011ef80, 0x4002573000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010038, 0x4002573000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40024aa240, 0x4002573000, 0x1000, 0x1000, 0x40024a2f00, 0x200000003, 0x40024a2f00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40000823c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000823c0, 0x40004ac00a, 0x40001998c8, 0x4d3b10, 0x40001ba060, 0x40004ac000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40000823c0, 0x0, 0x1, 0x0, 0x40001ba078, 0x40004ac000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004422cf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004422cf0, 0x4000210000, 0x1000, 0x1000, 0x40001999b8, 0x44d8a8, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400011d080, 0x4000210000, 0x1000, 0x1000, 0x40000826b8, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400011d080, 0x4000210000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400011d0c0, 0x4000210000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000199dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000199dd0, 0xa, 0x4000210201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000199dd0, 0x400021020a, 0x1, 0x40044152fb, 0x0, 0x4000210263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000199dd0, 0x400441e10a, 0x40044152fb, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40004fcad0, 0x40000426f0, 0x24, 0x4000199f20, 0x400007f5f0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x40004fcad0, 0x113075a, 0x7, 0x400007f5f0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x40004fca90) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 89 [IO wait]: internal/poll.runtime_pollWait(0xffffa1414b90, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011f618, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011f600, 0x4004478000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011f600, 0x4004478000, 0x1000, 0x1000, 0x40000ff688, 0x69f178, 0x40000ff6f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4004454060, 0x4004478000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40024aa6c0, 0x4004478000, 0x1000, 0x1000, 0x5, 0x40023ee500, 0x40000ff7c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004428780) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004428780, 0x68f90a, 0x400038b080, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004428780, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400043f5f0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400043f5f0, 0x4004492000, 0x1000, 0x1000, 0x40000ff998, 0x44e884, 0x40000ff948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40001d9980, 0x4004492000, 0x1000, 0x1000, 0x40000826b8, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40001d9980, 0x4004492000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001d9a00, 0x4004492000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40000ffd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000ffd80, 0xa, 0x4004492201, 0x1, 0xffff714ac130, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40000ffd80, 0x400449220a, 0x1, 0x400458e6d0, 0x0, 0x4004492259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40000ffd80, 0x40001e700a, 0x400458e6d0, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40004fcad0, 0x40024fc8c0, 0x1f, 0x40000ffee0, 0x40024fb230, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x40004fcad0, 0x113075a, 0x7, 0x40001b41f0, 0xd, 0x40024fb230, 0x0, 0x4714f4, 0x40004fcad0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x40004fca90, 0x40001b41f0, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 87 [select]: net/http.(*persistConn).readLoop(0x400000c900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 88 [select]: net/http.(*persistConn).writeLoop(0x400000c900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 108 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4004426300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 106 [select]: net/http.(*persistConn).readLoop(0x40024aa6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 72 [select]: net/http.(*persistConn).readLoop(0x40024aa7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 73 [select]: net/http.(*persistConn).writeLoop(0x40024aa7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 162 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x4004d90210) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 114 [IO wait]: internal/poll.runtime_pollWait(0xffffa14149c0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40004e8a98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40004e8a80, 0x4004742000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40004e8a80, 0x4004742000, 0x1000, 0x1000, 0x4000224be0, 0x469b10, 0x4000001680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001162b8, 0x4004742000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000384ea0, 0x4004742000, 0x1000, 0x1000, 0x4000224cf8, 0x67867c, 0x4000145ce0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400472c660) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x400472c660, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4000384ea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 115 [select]: net/http.(*persistConn).writeLoop(0x4000384ea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 111 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x4004426e00, 0x400441b8a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 112 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40002007e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 113 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40002007e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 130 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x40044363f0, 0x40045ca240) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 131 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x40044262c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 132 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x4004462a00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 133 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x4004462a00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 134 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x4004454298) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 135 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x4004462a00, 0x40045ca2a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 136 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000112be0, 0x1144559, 0x1b, 0x4000479e38, 0x40045ca3c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x400441bfc0, 0x40045ca3c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000479f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004429500, 0x400441bfc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 137 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 138 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x40044606e0, 0xd00007, 0x400458f900, 0x5, 0x400441bfd0, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 139 [IO wait]: internal/poll.runtime_pollWait(0xffffa1414f30, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004430a18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4004430a00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4004430a00, 0xffffa14cca68, 0x30, 0x4004bfa780) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004587ce0, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004587ce0, 0x4004bfa780, 0x11ae500, 0x40004a8400, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40004a8380, 0x12ed9d0, 0x4004587ce0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40004a8380, 0x12ed9d0, 0x4004587ce0, 0x40001fa700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 140 [IO wait]: internal/poll.runtime_pollWait(0xffffa14148d8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004430b18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4004430b00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4004430b00, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004587cf8, 0xffffa14ccf18, 0xffff714abcf0, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004587cf8, 0x665e1c, 0x0, 0xfe5e60, 0x400446d5c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004587d10, 0x40045c2d50, 0x11ae500, 0x40004a84e0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40004a8460, 0x12ec710, 0x4004587d10, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40004a8460, 0x12ec710, 0x4004587d10, 0x40001fa700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 129 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000493fc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 146 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40004fe7e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 120 [IO wait]: internal/poll.runtime_pollWait(0xffff71544700, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40004e9198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40004e9180, 0x4004ce8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40004e9180, 0x4004ce8000, 0x1000, 0x1000, 0x4000067be0, 0x469b10, 0x40045c5980) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000116460, 0x4004ce8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c66c0, 0x4004ce8000, 0x1000, 0x1000, 0x4000067cf8, 0x67867c, 0x400474d4a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400472d3e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x400472d3e0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40001c66c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 121 [select]: net/http.(*persistConn).writeLoop(0x40001c66c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 143 [select]: net/http.(*persistConn).writeLoop(0x40001c6fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 94 [IO wait]: internal/poll.runtime_pollWait(0xffffa1414aa8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011f798, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011f780, 0x4000211000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011f780, 0x4000211000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b8640, 0x4000211000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40024aa7e0, 0x4000211000, 0x1000, 0x1000, 0x0, 0x0, 0x4000489b28) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001bb2c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001bb2c0, 0xa, 0x0, 0x0, 0xffff7146e7d0, 0x0, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40001bb2c0, 0x413b8c, 0x30, 0x1063520, 0x400446cc01, 0x4ad6e4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004bfaff0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004bfaff0, 0x4004bfcc00, 0x200, 0x200, 0x4000100c00, 0x4000493940, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000493b00, 0x4004bfcc00, 0x200, 0x200, 0x40004a8380, 0x4000493940, 0x1135447) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000493b00, 0x4004bfcc00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000493b40, 0x4004bfcc00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004c4b4a0, 0x12ed900, 0x400446ceb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004c4b4a0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004c4b4a0, 0xee0d40, 0x4000181310, 0x12db068, 0x4000493b40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400466f4d0, 0x4004c50c60, 0x4004667800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 128 [IO wait]: internal/poll.runtime_pollWait(0xffff71544618, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000209698, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000209680, 0x4004d73000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000209680, 0x4004d73000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000104e0, 0x4004d73000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c7200, 0x4004d73000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004c51ec0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004c51ec0, 0x100a, 0xf1f980, 0x0, 0x37, 0x0, 0x40000f1c38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004c51ec0, 0x0, 0x40000f1c78, 0x655784, 0x40000f1ca8, 0x40b2f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004cdf440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004cdf440, 0x4004ca5c00, 0x200, 0x200, 0x4, 0x0, 0x4000097901) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400472b640, 0x4004ca5c00, 0x200, 0x200, 0x4004cac607, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400472b640, 0x4004ca5c00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400472b680, 0x4004ca5c00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004e2f1e0, 0x680d00, 0x40000f1eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004e2f1e0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004e2f1e0, 0xee0d40, 0x400012afa0, 0x12db068, 0x400472b680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004cc73b0, 0x400472de00, 0x4004e32660) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 147 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40004fe7e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 148 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40004fe7e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 149 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40004fe7e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 150 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4000493f80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 142 [IO wait]: internal/poll.runtime_pollWait(0xffff715447e8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000209098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000209080, 0x40045f0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000209080, 0x40045f0000, 0x1000, 0x1000, 0x40045eabe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40044542b8, 0x40045f0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c6fc0, 0x40045f0000, 0x1000, 0x1000, 0x40045eacf8, 0x67867c, 0x40045ca5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40044296e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40044296e0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40001c6fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 153 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004d90370, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 154 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004d90370, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 155 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4004d90370) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 156 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40004fe7e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 157 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000112be0, 0x1144559, 0x1b, 0x4004739e38, 0x4004da4120, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004671af0, 0x4004da4120, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004739f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004c51200, 0x4004671af0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 160 [select]: net/http.(*persistConn).readLoop(0x40001c7200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 161 [select]: net/http.(*persistConn).writeLoop(0x40001c7200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 163 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x4004d90210) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 164 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x4004d90210) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 165 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x4004d90210) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 166 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004ce7300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 167 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x4004ce7300, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 168 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x40001fa700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 169 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x40001fa700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 170 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x40001fa700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 171 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:48:37.667+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:48:37.667+00:00 [Info] Setting ipv6=false 2021-08-16T13:48:37.689+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:48:37.691+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:48:37.691+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:48:37.691+00:00 [Info] Setting log level to Info 2021-08-16T13:48:37.691+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:48:37.693+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:48:37 audit: created new audit service 2021-08-16T13:48:37.702+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:48:37.702+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:48:37.702+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:48:37.702+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:48:37.702+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:48:37.713+00:00 [Info] security context initialized 2021-08-16T13:48:37.713+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:48:37.714+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:48:37.714+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:48:37.714+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:48:37.714+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:48:37.728+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:48:37.728+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:48:37.729+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:48:37.748+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:48:37.750+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:48:37.752+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:48:37.752+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:48:37.755+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:48:37.755+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:48:37.767+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:48:37.769+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:48:37.811+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:48:37.811+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 58 us 2021-08-16T13:48:37.811+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:48:37.813+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:48:37.814+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:48:37.818+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:48:37.818+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:48:37.818+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:48:37.818+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:48:37.818+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:48:37.818+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:37.818+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:37.820+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:48:37.820+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:48:37.820+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:48:37.820+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:37.820+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:48:37.820+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:48:37.820+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:48:37.820+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:48:37.820+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:48:37.820+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:48:37.820+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:48:37.825+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 5.417019ms 2021-08-16T13:48:37.826+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:48:37.827+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:48:37.827+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:37.827+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:48:37.827+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:48:37.827+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:48:37.827+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:48:37.837+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:48:37.838+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:48:37.838+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:48:37.838+00:00 [Info] RebalanceMgr::initService Init 2021-08-16T13:48:37.838+00:00 [Info] ServiceMgr::registerWithServer nodeuuid 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:37.838+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:48:37.839+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x4004e1691f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x4004e1691f pc=0xbd8ea4] goroutine 185 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004e6c9f0 sp=0x4004e6c9c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004e6ca30 sp=0x4004e6c9f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x4004e54340, 0x4004b83200, 0x11a9890, 0x4004e6cb40, 0x4004e54358, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004e6cab0 sp=0x4004e6ca40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x4004e54340, 0x4004b83200, 0x11a9890, 0x4004e6cb40, 0x4004e54358, 0xc18f8c, 0x4004e6cc18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004e6caf0 sp=0x4004e6cab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x40046fe3c0, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004e6cda0 sp=0x4004e6caf0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x40046fe3c0, 0x40004b9810, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004e6ce60 sp=0x4004e6cda0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004e564d0, 0x4004e564e0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004e6d200 sp=0x4004e6ce60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004e564d0, 0x4004e564e0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004e6d8f0 sp=0x4004e6d200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004e3d850, 0x4004e0d180, 0x40046fab00, 0x4004e564b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004e6dfb0 sp=0x4004e6d8f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004e6dfb0 sp=0x4004e6dfb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x40003f4000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x4000102c60, 0x4, 0x1136a87, 0xf, 0x400018be68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 7 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400000e678) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 8 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400000e690) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 9 [IO wait]: internal/poll.runtime_pollWait(0xffff680c79d8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f8898, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f8880, 0x4000225000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f8880, 0x4000225000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010588, 0x4000225000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x400008aa20, 0x4000456001, 0xdff, 0xdff, 0x40000a98e8, 0x6902d8, 0x4000187948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400000e6d8, 0x4000456001, 0xdff, 0xdff, 0x40001879f8, 0x6f1968, 0x4000187938) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40000d6b00, 0xa, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40000d6b00, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40000d6b00, 0xf5a960, 0x400008ab00, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x400008aae0, 0x400010a260, 0x4000187b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x4000094820, 0x12f8f98, 0x400008aae0, 0x400010cb87, 0x16, 0x4000410360, 0x40000a9c01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x4000094820, 0x12f8f98, 0x400008aae0, 0x4000094820, 0x4000036c28, 0x4000036c30, 0x40000f8900, 0x4000046660, 0xee0980, 0x4000410360, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x4000094820, 0x12f8f98, 0x400008aae0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x400009ff50, 0x40000a9f78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x40000a9f78, 0x400009ff50, 0x12db8a8, 0x400000e6a8, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x400009ff50, 0x40000b9220, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x400009ff50, 0x40000b9220) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 18 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 19 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 20 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 21 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 22 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 34 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400019c000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 23 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40000547a0, 0x4, 0x4000054668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x400008a060, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 27 [IO wait]: internal/poll.runtime_pollWait(0xffff680c78f0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000136698, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000136680, 0x4000457000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000136680, 0x4000457000, 0x1000, 0x1000, 0x4000065a88, 0x68f7a4, 0x400046e050) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000190028, 0x4000457000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c5a0, 0x4000457000, 0x1000, 0x1000, 0xf12180, 0x40003f0220, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003be600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003be600, 0xa, 0x1008880, 0x40003f01e0, 0x199, 0x1000, 0x4000065b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003be600, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400045c1e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400045c1e0, 0x4000472000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400046a000, 0x4000472000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400046a000, 0x4000472000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400046a040, 0x4000472000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x400046e000, 0x0, 0x4000065eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400046e000, 0x0, 0x0, 0x4000065f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400046e000, 0xee0d40, 0x40003f01e0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40000e6090, 0x400008a4e0, 0x400046c000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 43 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000081fc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 37 [select]: net/http.(*persistConn).readLoop(0x400000c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 38 [select]: net/http.(*persistConn).writeLoop(0x400000c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 11 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000106340, 0x12c18bc, 0x1, 0x40000a7e58, 0x400012e7e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x40001a1ec8, 0x400012e7e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40000a7fa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x400010e000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 12 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x400010e000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 66 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x40000ee7e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 15 [IO wait]: internal/poll.runtime_pollWait(0xffff680c7808, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000136798, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000136780, 0x4000246000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000136780, 0x4000246000, 0x1000, 0x1000, 0x4000245be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000108180, 0x4000246000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000cc60, 0x4000246000, 0x1000, 0x1000, 0x4000245cf8, 0x67867c, 0x4000038840) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000180a80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000180a80, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000cc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 16 [select]: net/http.(*persistConn).writeLoop(0x400000cc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 41 [select]: net/http.(*persistConn).readLoop(0x40004107e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 42 [select]: net/http.(*persistConn).writeLoop(0x40004107e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 74 [IO wait]: internal/poll.runtime_pollWait(0xffff680c71b0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004592098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004592080, 0x40044fd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004592080, 0x40044fd000, 0x1000, 0x1000, 0x40, 0x1d, 0x400458f7b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40004f4050, 0x40044fd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004590000, 0x40044fd000, 0x1000, 0x1000, 0x400019ec00, 0x200000003, 0x400019ec00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40004bc720) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40004bc720, 0x400024a00a, 0x400458f8c8, 0x4d3b10, 0x400008a0c0, 0x400024a000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40004bc720, 0x0, 0x1, 0x4dd090, 0x400008a0d8, 0x400024a000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40004aca50) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40004aca50, 0x40045a1000, 0x1000, 0x1000, 0x400458f9b8, 0x10, 0x400458fa68) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40004b65c0, 0x40045a1000, 0x1000, 0x1000, 0x4000488cb8, 0x58, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40004b65c0, 0x40045a1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40004b6600, 0x40045a1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400458fdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400458fdd0, 0xa, 0x40045a1201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400458fdd0, 0x40045a120a, 0x1, 0x40004a7a49, 0x0, 0x40045a12e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400458fdd0, 0x400448280a, 0x40004a7a49, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4002486ee0, 0x40004b4198, 0x17, 0x400458ff20, 0x40044f8558, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x4002486ee0, 0x113075a, 0x7, 0x40044f8558, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x4002486ea0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 121 [select]: net/http.(*persistConn).writeLoop(0x400000d7a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 77 [select]: net/http.(*persistConn).readLoop(0x40024d6900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 75 [IO wait]: internal/poll.runtime_pollWait(0xffff680c7298, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f8b18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f8b00, 0x4000015000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f8b00, 0x4000015000, 0x1000, 0x1000, 0x40, 0x1d, 0x400022b7b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000190230, 0x4000015000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40004107e0, 0x4000015000, 0x1000, 0x1000, 0x4000202480, 0x200000003, 0x4000202480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003bf440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003bf440, 0x400024a00a, 0x400022b8c8, 0x4d3b10, 0x400008a0c0, 0x400024a000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003bf440, 0x0, 0x1, 0x4dd090, 0x400008a0d8, 0x400024a000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40004ac3f0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40004ac3f0, 0x40044fc000, 0x1000, 0x1000, 0x400022b9b8, 0x0, 0x400022b988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40004b6340, 0x40044fc000, 0x1000, 0x1000, 0x400019e480, 0x3, 0x400022bbe0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40004b6340, 0x40044fc000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40004b6380, 0x40044fc000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400022bdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400022bdd0, 0xa, 0x40044fc201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400022bdd0, 0x40044fc20a, 0x1, 0x40004a75cb, 0x0, 0x40044fc263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400022bdd0, 0x40004ba10a, 0x40004a75cb, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4002486ee0, 0x4000043920, 0x24, 0x400022bf20, 0x400451b818, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x4002486ee0, 0x113075a, 0x7, 0x400451b818, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x4002486ea0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 78 [select]: net/http.(*persistConn).writeLoop(0x40024d6900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 92 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x400021b900) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 69 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x40000ee7e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 70 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x40000ee7e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 71 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x40000ee7e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 91 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x400021b900) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 59 [IO wait]: internal/poll.runtime_pollWait(0xffff680c6ef8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000137398, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000137380, 0x4004525000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000137380, 0x4004525000, 0x1000, 0x1000, 0x1, 0x0, 0x40002296f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000108498, 0x4004525000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40024d6900, 0x4004525000, 0x1000, 0x1000, 0x5, 0x40023f6f00, 0x40002297c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000488f00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000488f00, 0x68f90a, 0x40001dba20, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000488f00, 0x0, 0x2, 0x0, 0x1, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004450c30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004450c30, 0x4004637000, 0x1000, 0x1000, 0x4000229998, 0x44e884, 0x4000229a18) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400463e080, 0x4004637000, 0x1000, 0x1000, 0x4000488cb8, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400463e080, 0x4004637000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400463e0c0, 0x4004637000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000229d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000229d80, 0xa, 0x4004637201, 0x1, 0xffff6802d4b0, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000229d80, 0x400463720a, 0x1, 0x400443f8b0, 0x0, 0x4004637259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000229d80, 0x400444340a, 0x400443f8b0, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4002486ee0, 0x40044fa560, 0x1f, 0x4000229ee0, 0x40044f8c60, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x4002486ee0, 0x113075a, 0x7, 0x40004a7a60, 0xd, 0x40044f8c60, 0x0, 0x0, 0x2) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x4002486ea0, 0x40004a7a60, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 53 [select]: net/http.(*persistConn).readLoop(0x4004590000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 54 [select]: net/http.(*persistConn).writeLoop(0x4004590000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 33 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400046ab40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 101 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400046b280) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 98 [select]: net/http.(*persistConn).writeLoop(0x4004590360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 45 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40024803c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 46 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x40044105a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 47 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x4002449890) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 48 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x40000f9500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 49 [IO wait]: internal/poll.runtime_pollWait(0xffff680c6e10, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f9698, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f9680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f9680, 0x94fbb0, 0x4000055e98, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004437a70, 0xffff91d53a68, 0x0, 0x400443a820) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004437a70, 0x12ed9d0, 0x4004437a70, 0xf1f4c0, 0x40000b9201) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x40000f9600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 82 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400244d520, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 83 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400244d520, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 84 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x400244d520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 85 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x4004450510) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 93 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x400463fbc0, 0x4004c2a5d0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 89 [select]: net/http.(*persistConn).readLoop(0x4004438480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 88 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x4000218380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 90 [select]: net/http.(*persistConn).writeLoop(0x4004438480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 32 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400463e340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 81 [select]: net/http.(*persistConn).readLoop(0x4004590360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 65 [IO wait]: internal/poll.runtime_pollWait(0xffff593224c8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004592a18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004592a00, 0x40045c6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004592a00, 0x40045c6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010460, 0x40045c6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40045906c0, 0x40045c6000, 0x1000, 0x1000, 0x60, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40004bd9e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40004bd9e0, 0x1, 0x4004481b60, 0x43d584, 0x0, 0x0, 0x4004e0f568) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40045906c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 195 [select]: net/http.(*persistConn).roundTrip(0x400000d7a0, 0x4000081dc0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004c2f100, 0xf8, 0x10a8660, 0x4004d3a588) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004c2f100, 0x1b6e900, 0xc03eb92b71fee406, 0x1bfb1640e9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004c2f000, 0x12db048, 0x1b6e900, 0xc03eb92b71fee406, 0x1bfb1640e9, 0x1be4a80, 0x0, 0x40001907a8, 0x4004c5ffb0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004c5ff80, 0x4004c2f000, 0xc03eb92b71fee406, 0x1bfb1640e9, 0x1be4a80, 0x40001907a8, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004c5ff80, 0x4004c2f000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x40001deff0, 0x1b, 0x4004d3acf0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004c355f0, 0x112f905, 0x6, 0x12db908, 0x4004e90020, 0xee40c0, 0x4004c2cc18, 0x4004d3acf0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x40045ac280, 0x46, 0x12db908, 0x4004e90020, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x40045ac280, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x4004c5fec0, 0xffff59350430) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004d554b0, 0x1063520, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x4004c706c0, 0x4004c706c0, 0x40045ac280) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.FetchNewClusterInfoCache(0xffffdd22594e, 0xe, 0x113075a, 0x7, 0x1130e44, 0x8, 0x40045d20c4, 0x2, 0xf01) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:123 +0x134 github.com/couchbase/indexing/secondary/manager.(*IndexManager).FetchNewClusterInfoCache(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:306 github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).getIndexMetadata(0x1be5520, 0x12ec7a0, 0x4000081d80, 0x4000095e00, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:1284 +0x74 github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).handleIndexMetadataRequest(0x1be5520, 0x12edbe0, 0x40001bafc0, 0x4004521100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:1271 +0x3e8 net/http.HandlerFunc.ServeHTTP(0x4004c2aff0, 0x12edbe0, 0x40001bafc0, 0x4004521100) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2069 +0x40 net/http.(*ServeMux).ServeHTTP(0x400463fec0, 0x12edbe0, 0x40001bafc0, 0x4004521100) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2448 +0x190 net/http.serverHandler.ServeHTTP(0x40001bad20, 0x12edbe0, 0x40001bafc0, 0x4004521100) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2887 +0xbc net/http.(*conn).serve(0x4004440820, 0x12f86a0, 0x4002480f00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:1952 +0x71c created by net/http.(*Server).Serve /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:3013 +0x318 goroutine 102 [IO wait]: internal/poll.runtime_pollWait(0xffff680c70c8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400047c618, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400047c600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400047c600, 0x58ec48, 0x4004481140, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004bec018, 0x4714f4, 0x4004481140, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004bec018, 0x400443f800, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x40004b7100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 103 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x400463fbc0, 0x40004b8a10, 0x40004b8a80, 0x4004c2a5d0, 0x12e8f90, 0x40001849a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x40046d1cc4, 0x5, 0x40004b7100, 0x12eca70, 0x40004b6fc0, 0x130d478, 0x4000114a10, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40000ee380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40000ee380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 94 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40001849a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 95 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40001849a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 96 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x4004640630, 0x4000039b00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 97 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x400463e300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 130 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x400021b9a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 131 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x400021b9a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 132 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x40001906a8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 133 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x400021b9a0, 0x4000039bc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 134 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000106340, 0x1144559, 0x1b, 0x4004d07e38, 0x4000039d40, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004c2b0f0, 0x4000039d40, 0xf1f700, 0x4000107bc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004d07f80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004c7a780, 0x4004c2b0f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 135 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 136 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x40000954a0, 0xd00007, 0x4004c27978, 0x5, 0x4004c2b100, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 137 [IO wait]: internal/poll.runtime_pollWait(0xffff680c6c40, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004c2c618, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4004c2c600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4004c2c600, 0x0, 0x4000242d88, 0x665fe4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c23cf8, 0x611a6cb5, 0x4000242db8, 0x445c24) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c23cf8, 0x4000242e20, 0x4000242e28, 0x18, 0x400447b500) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40001bad20, 0x12ed9d0, 0x4004c23cf8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40001bad20, 0x12ed9d0, 0x4004c23cf8, 0x40003f4000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 138 [IO wait]: internal/poll.runtime_pollWait(0xffff680c6b58, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004c2c718, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4004c2c700, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4004c2c700, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c23d10, 0xffff91d53a68, 0xffff59350760, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c23d10, 0x665e1c, 0x0, 0xfe5e60, 0x4004c745c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004c23d28, 0x4004c5ed50, 0x11ae500, 0x40001bae80, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40001bae00, 0x12ec710, 0x4004c23d28, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40001bae00, 0x12ec710, 0x4004c23d28, 0x40003f4000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 119 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).processSchedIndexes(0x40024e78c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:296 +0x44 created by github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:218 +0x3c goroutine 147 [IO wait]: internal/poll.runtime_pollWait(0xffff680c6fe0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004592818, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004592800, 0x4004699000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004592800, 0x4004699000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001080d8, 0x4004699000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004590360, 0x4004699000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400008ae40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400008ae40, 0xa, 0x0, 0x0, 0x0, 0x0, 0x4004c74438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400008ae40, 0x412bc0, 0x4004c74478, 0x655784, 0x4004c744c8, 0x412bc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004c5f7a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004c5f7a0, 0x400475bc00, 0x200, 0x200, 0x4, 0x0, 0x4004c74578) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000081780, 0x400475bc00, 0x200, 0x200, 0x40045ac207, 0x33, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000081780, 0x400475bc00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000081800, 0x400475bc00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004c33600, 0x12ec700, 0x4004c746b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004c33600, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004c33600, 0xee0d40, 0x4000095ae0, 0x12db068, 0x4000081800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004c34cf0, 0x4004c7b3e0, 0x4004d18780) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 145 [IO wait]: internal/poll.runtime_pollWait(0xffff593225b0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004c2ca98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004c2ca80, 0x4004d56000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004c2ca80, 0x4004d56000, 0x1000, 0x1000, 0x400462abe0, 0x469b10, 0x400447bb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000190740, 0x4004d56000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000d440, 0x4004d56000, 0x1000, 0x1000, 0x400462acf8, 0x67867c, 0x4004d18720) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004c7b260) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004c7b260, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000d440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 146 [select]: net/http.(*persistConn).writeLoop(0x400000d440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 149 [IO wait]: internal/poll.runtime_pollWait(0xffff593223e0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004592b18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004592b00, 0x40045d1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004592b00, 0x40045d1000, 0x1000, 0x1000, 0x0, 0x40045d0000, 0x400022fa38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000104a0, 0x40045d1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).fill(0x40004bdaa0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40004bdaa0, 0x654f0a, 0x400022fb08, 0x439548, 0x400022faf8, 0x5429e4, 0x4004592b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).ReadLine(0x40004bdaa0, 0x655058, 0x400022fb08, 0x655070, 0x400022fbc8, 0x412ff8, 0x40004a5e60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:389 +0x30 net/textproto.(*Reader).readLineSlice(0x400022fcf0, 0x40004a5e60, 0x400022fc48, 0x6567e4, 0x8b33a12aaccc3ac6, 0xf26100) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/textproto/reader.go:57 +0xac net/textproto.(*Reader).ReadLine(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/textproto/reader.go:38 net/http.ReadResponse(0x40004bdaa0, 0x4004521000, 0x1000, 0x40045d1000, 0x40000104a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/response.go:161 +0x8c github.com/couchbase/cbauth/revrpc.(*Service).Run(0x40004ad560, 0x400022ff10, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:132 +0x2ac github.com/couchbase/cbauth/revrpc.BabysitService(0x400022ff10, 0x40004ad560, 0x0, 0x0, 0x0, 0xd8d03c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth/service.RegisterManager(0x13007d8, 0x400047bf00, 0x0, 0x0, 0x40045cef80, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:146 +0xa0 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).registerWithServer(0x400047bf00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:301 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:205 +0x78 goroutine 112 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40000ee620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 111 [semacquire]: sync.runtime_Semacquire(0x4004e3d858) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004e3d850) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x40046fab00, 0x4000204480, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffdd225969, 0x29, 0xffffdd2259cb, 0x25, 0x4004e1aea0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004bf09b0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffdd225969, 0x29, 0xffffdd2259cb, 0x25, 0x4004e1aea0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004bf09b0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004e6d190, 0x4004e6ce60, 0x4000204480, 0x400038f8c0, 0x4000081fc0, 0x0, 0x17f, 0x4004d00da8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x40003f4000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004bf09b0, 0xb, 0x4004bf09a8, 0x6, 0x4004bf09c0, 0xd, 0x4004be4ca0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x40003f4000, 0x40003b0000, 0x40001821b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x40003f4000, 0x400443a740, 0x2, 0x2, 0x400443a750, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 118 [IO wait]: internal/poll.runtime_pollWait(0xffff59322698, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400047cc98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400047cc80, 0x4004e0b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400047cc80, 0x4004e0b000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000108638, 0x4004e0b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001119e0, 0x4004e0b000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004e18900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004e18900, 0xa, 0x0, 0x0, 0x0, 0x0, 0x400459c438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004e18900, 0x0, 0x400459c478, 0x655784, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000205530) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000205530, 0x400242ea00, 0x200, 0x200, 0x4, 0x0, 0x400459c508) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400046b780, 0x400242ea00, 0x200, 0x200, 0x4000105cb7, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400046b780, 0x400242ea00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400046b7c0, 0x400242ea00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004d9c9a0, 0x0, 0x400459c6b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004d9c9a0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004d9c9a0, 0xee0d40, 0x400019adc0, 0x12db068, 0x400046b7c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400027d170, 0x4002416840, 0x4004d9e480) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 113 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40000ee620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 162 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40000ee620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 163 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40000ee620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 164 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x40004b7880) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 165 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40024e7a20, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 166 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40024e7a20, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 167 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x40024e7a20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 168 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40000ee620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 169 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000106340, 0x1144559, 0x1b, 0x400458de38, 0x40046b9b00, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4000461b30, 0x40046b9b00, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400458df80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x400008bf20, 0x4000461b30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 174 [runnable]: net/http.(*persistConn).roundTrip(0x4004438480, 0x400046bb40, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004dc0700, 0xf8, 0x10a8660, 0x400458ae18) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004dc0700, 0x1b6e900, 0xc03eb92b71f27db0, 0x1bfb09da93) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004dc0600, 0x12db048, 0x1b6e900, 0xc03eb92b71f27db0, 0x1bfb09da93, 0x1be4a80, 0x0, 0x40004f4328, 0x4004f1a270, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004f1a240, 0x4004dc0600, 0xc03eb92b71f27db0, 0x1bfb09da93, 0x1be4a80, 0x40004f4328, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004f1a240, 0x4004dc0600, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004d98990, 0x1b, 0x400458b580, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004dd4510, 0x112f905, 0x6, 0x12db908, 0x4000492e20, 0xee40c0, 0x40000f9d18, 0x400458b580, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x400003e700, 0x46, 0x12db908, 0x4000492e20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x400003e700, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x20, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400458bd40, 0x4, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x4004e16000, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x4004e16000, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40004b78c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1267 +0xf8 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 172 [select]: net/http.(*persistConn).readLoop(0x40001119e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 173 [select]: net/http.(*persistConn).writeLoop(0x40001119e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 175 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x40024e78c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:224 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 176 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x40024e78c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:612 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 177 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x40024e78c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:662 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 178 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x40024e78c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:752 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 179 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x400047bf00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:249 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 155 [IO wait]: internal/poll.runtime_pollWait(0xffff59322210, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004592b98, 0x72, 0x0, 0x1, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004592b80, 0x40004ad781, 0x1, 0x1, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004592b80, 0x40004ad781, 0x1, 0x1, 0x0, 0x4714f4, 0x4004d18e40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000104e8, 0x40004ad781, 0x1, 0x1, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*connReader).backgroundRead(0x40004ad770) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:692 +0x50 created by net/http.(*connReader).startBackgroundRead /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:688 +0xc4 goroutine 181 [select]: net/http.(*persistConn).roundTrip(0x40045906c0, 0x4002480e40, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004520f00, 0xf8, 0x10a8660, 0x4004588dd8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004520f00, 0x1b6e900, 0xc03eb92b71f8d707, 0x1bfb1033f3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004520e00, 0x12db048, 0x1b6e900, 0xc03eb92b71f8d707, 0x1bfb1033f3, 0x1be4a80, 0x0, 0x4000010410, 0x40004ad350, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x40004ad320, 0x4004520e00, 0xc03eb92b71f8d707, 0x1bfb1033f3, 0x1be4a80, 0x4000010410, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x40004ad320, 0x4004520e00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x40000426c0, 0x1b, 0x4004589540, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x40004a59e0, 0x112f905, 0x6, 0x12db908, 0x400010a540, 0xee40c0, 0x4004592918, 0x4004589540, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x40045c4000, 0x46, 0x12db908, 0x400010a540, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x40045c4000, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x240, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004589d00, 0x40004ad260, 0xffff680c0af8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x40045b6240, 0xd47600, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x40045b6240, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x40003f4000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9702 +0x344 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 182 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x40003f4000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1090 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 183 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x40003f4000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1017 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 184 [sleep]: time.Sleep(0x12a05f200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/plasma.RunMemQuotaTunerIteration(0x400462dfb0, 0xdf74d8, 0x400012e660) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:211 +0x24c github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:247 +0xc0 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 goroutine 194 [select]: net/http.(*persistConn).writeLoop(0x40045906c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 150 [select]: net/http.(*persistConn).roundTrip(0x400000d680, 0x4000081bc0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004c2ef00, 0xf8, 0x10a8660, 0x4004e24798) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004c2ef00, 0x1b6e900, 0xc03eba3971fbdff2, 0x11770176cdd) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004c2ee00, 0x12db048, 0x1b6e900, 0xc03eba3971fbdff2, 0x11770176cdd, 0x1be4a80, 0x0, 0x4000190788, 0x4004c5fbf0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004c5fbc0, 0x4004c2ee00, 0xc03eba3971fbdff2, 0x11770176cdd, 0x1be4a80, 0x4000190788, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004c5fbc0, 0x4004c2ee00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004d7bb40, 0x1f, 0x4004e24d10, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/indexer.getWithAuth(0x4004d7bb40, 0x1f, 0xe, 0x113846e, 0x11) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:3077 +0x68 github.com/couchbase/indexing/secondary/indexer.getGlobalTopology(0x4004e3ce70, 0xe, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:2266 +0x78 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).updateNodeList(0x400047bf00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:220 +0x3c created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:206 +0x98 goroutine 153 [IO wait]: internal/poll.runtime_pollWait(0xffff593222f8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004c2cb98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004c2cb80, 0x4004e92000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004c2cb80, 0x4004e92000, 0x1000, 0x1000, 0x4004c78be8, 0x7ff1a8, 0x4004c78cb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000190798, 0x4004e92000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000d680, 0x4004e92000, 0x1000, 0x1000, 0x60, 0x0, 0x4004d7bb01) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004c7bc80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004c7bc80, 0x1, 0x4004d18f60, 0x43d584, 0x0, 0xd8c91c, 0x4004c78d68) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000d680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 154 [select]: net/http.(*persistConn).writeLoop(0x400000d680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 120 [IO wait]: internal/poll.runtime_pollWait(0xffff59322128, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004c2cd18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004c2cd00, 0x4004dcd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004c2cd00, 0x4004dcd000, 0x1000, 0x1000, 0x40001a1448, 0x7356e4, 0xffffdd22594e) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40004f4330, 0x4004dcd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000d7a0, 0x4004dcd000, 0x1000, 0x1000, 0x60, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002416e40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4002416e40, 0x1, 0x4004d9ef00, 0xe, 0x1c1c218, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000d7a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c Initializing write barrier = 8000 2021-08-16T13:48:43.000+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:48:43.001+00:00 [Info] Setting ipv6=false 2021-08-16T13:48:43.021+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:48:43.024+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:48:43.024+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:48:43.024+00:00 [Info] Setting log level to Info 2021-08-16T13:48:43.024+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:48:43.027+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:48:43 audit: created new audit service 2021-08-16T13:48:43.036+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:48:43.036+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:48:43.036+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:48:43.036+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:48:43.036+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:48:43.047+00:00 [Info] security context initialized 2021-08-16T13:48:43.047+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:48:43.047+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:48:43.047+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:48:43.047+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:48:43.047+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:48:43.061+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:48:43.062+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:48:43.062+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:48:43.089+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:48:43.090+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:48:43.090+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:48:43.092+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:48:43.093+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:48:43.093+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:48:43.093+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:48:43.095+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:48:43.144+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:48:43.144+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 109 us 2021-08-16T13:48:43.145+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:48:43.147+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:48:43.147+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:48:43.151+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:48:43.151+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:48:43.151+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:48:43.151+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:48:43.151+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:48:43.151+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:43.151+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:43.153+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:48:43.153+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:48:43.153+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:48:43.153+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:43.153+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:48:43.153+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:48:43.153+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:48:43.153+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:48:43.153+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:48:43.153+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:48:43.153+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:48:43.160+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 6.848089ms 2021-08-16T13:48:43.160+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:48:43.162+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:48:43.162+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:43.162+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:48:43.162+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:48:43.162+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:48:43.162+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:48:43.172+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:48:43.172+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:48:43.172+00:00 [Info] RebalanceMgr::initService Init 2021-08-16T13:48:43.173+00:00 [Info] ServiceMgr::registerWithServer nodeuuid 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:43.173+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:48:43.173+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:48:43.174+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x4004d8ab5f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x4004d8ab5f pc=0xbd8ea4] goroutine 183 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004eda9f0 sp=0x4004eda9c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004edaa30 sp=0x4004eda9f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x4004eb2340, 0x4004b1ac00, 0x11a9890, 0x4004edab40, 0x4004eb2358, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004edaab0 sp=0x4004edaa40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x4004eb2340, 0x4004b1ac00, 0x11a9890, 0x4004edab40, 0x4004eb2358, 0xc18f8c, 0x4004edac18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004edaaf0 sp=0x4004edaab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4000033400, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004edada0 sp=0x4004edaaf0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4000033400, 0x400237d8f0, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004edae60 sp=0x4004edada0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004eb0580, 0x4004eb0590, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004edb200 sp=0x4004edae60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004eb0580, 0x4004eb0590, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004edb8f0 sp=0x4004edb200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004ddfc80, 0x4004eba700, 0x40000d2680, 0x4004eb0560) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004edbfb0 sp=0x4004edb8f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004edbfb0 sp=0x4004edbfb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x4000158380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x40000fb5c0, 0x4, 0x1136a87, 0xf, 0x40000f9e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 7 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400000e690) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 8 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400000e6a8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 9 [IO wait]: internal/poll.runtime_pollWait(0xffff74134198, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0898, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0880, 0x40001a3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0880, 0x40001a3000, 0x1000, 0x1000, 0x1311860, 0x47b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000105b0, 0x40001a3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x4000082cc0, 0x4000136001, 0xdff, 0xdff, 0x400009f8e8, 0x6902d8, 0x40000f5948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400000e6f0, 0x4000136001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x400012e000, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400012e000, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400012e000, 0xf5a960, 0x4000124080, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4000124060, 0x40001300e0, 0x40000f5b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400011e000, 0x12f8f98, 0x4000124060, 0x400010e867, 0x16, 0x400013a120, 0x400009fc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400011e000, 0x12f8f98, 0x4000124060, 0x400011e000, 0x400010e070, 0x400010e080, 0x400012c000, 0x4000130000, 0xee0980, 0x400013a120, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400011e000, 0x12f8f98, 0x4000124060) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x40000fa3c0, 0x400009ff78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400009bf78, 0x40000fa3c0, 0x12db8a8, 0x400000e6c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x40000fa3c0, 0x40000b1540, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x40000fa3c0, 0x40000b1540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 19 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400010c100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 10 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 11 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 12 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 13 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 14 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 15 [syscall]: syscall.Syscall(0x3f, 0x0, 0x4000377fa0, 0x4, 0x4000377e68, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x4000377fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x4000377fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x4000377fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 58 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x40013d4000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 35 [select]: net/http.(*persistConn).readLoop(0x400013a000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 36 [select]: net/http.(*persistConn).writeLoop(0x400013a000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 50 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000196450, 0x12c18bc, 0x1, 0x400009de58, 0x400016a6c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x40003726c8, 0x400016a6c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400009dfa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4000120180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 51 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4000120180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 83 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x40003ca360) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 54 [IO wait]: internal/poll.runtime_pollWait(0xffff74133fc8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400012c218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400012c200, 0x40013b6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400012c200, 0x40013b6000, 0x1000, 0x1000, 0x4000014be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001280e0, 0x40013b6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400031c360, 0x40013b6000, 0x1000, 0x1000, 0x4000014cf8, 0x67867c, 0x400016ac00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40013b4000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40013b4000, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400031c360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 55 [select]: net/http.(*persistConn).writeLoop(0x400031c360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 24 [IO wait]: internal/poll.runtime_pollWait(0xffff741340b0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0080, 0x4000137000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0080, 0x4000137000, 0x1000, 0x1000, 0x4000019a88, 0x68f7a4, 0x40000f2050) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000100b8, 0x4000137000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400013a000, 0x4000137000, 0x1000, 0x1000, 0xf12180, 0x40000f4040, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400032f4a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400032f4a0, 0xa, 0x1008880, 0x40000f4000, 0x199, 0xf03, 0x4000019b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400032f4a0, 0x2, 0x0, 0x9, 0x9b5, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000114120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000114120, 0x40000f8000, 0xe00, 0xe00, 0x45c, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000080000, 0x40000f8000, 0xe00, 0xe00, 0x400, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000080000, 0x40000f8000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000080140, 0x40000f8000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40000f2000, 0x0, 0x4000019eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40000f2000, 0x0, 0x0, 0x4000019f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40000f2000, 0xee0d40, 0x40000f4000, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40000de000, 0x4000124780, 0x400001e000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 81 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x4000303f80, 0x4004b00830) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 66 [select]: net/http.(*persistConn).readLoop(0x400000cd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 67 [select]: net/http.(*persistConn).writeLoop(0x400000cd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 27 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40003b0540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 82 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001203c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 84 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x40003ba600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 85 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x4002410180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 125 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40013d5260) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 61 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x40013d4000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 62 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x40013d4000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 63 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x40013d4000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 124 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40013d5260) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 86 [IO wait]: internal/poll.runtime_pollWait(0xffff74133970, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4002410318, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4002410300, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4002410300, 0x0, 0x4000054698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40003b6528, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40003b6528, 0x12ed9d0, 0x40003b6528, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x4002410280) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 87 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400248a8f0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 88 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400248a8f0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 89 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x400248a8f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 90 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x40003bb590) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 96 [IO wait]: internal/poll.runtime_pollWait(0xffff741336b8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4002410518, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4002410500, 0x40013fa000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4002410500, 0x40013fa000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40003d8170, 0x40013fa000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400250cb40, 0x40013fa000, 0x1000, 0x1000, 0x4001396f00, 0x200000003, 0x4001396f00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003cb380) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003cb380, 0x40013bc00a, 0x40024b18c8, 0x4d3b10, 0x40000820c0, 0x40013bc000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003cb380, 0x0, 0x1, 0x0, 0x40000820d8, 0x40013bc000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400037c480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400037c480, 0x40024ba000, 0x1000, 0x1000, 0x40024b19b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000302380, 0x40024ba000, 0x1000, 0x1000, 0x4000082898, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000302380, 0x40024ba000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40003023c0, 0x40024ba000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40024b1dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40024b1dd0, 0xa, 0x40024ba201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40024b1dd0, 0x40024ba20a, 0x1, 0x40044c25f9, 0x0, 0x40024ba2e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40024b1dd0, 0x40034cc90a, 0x40044c25f9, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400248aa00, 0x40003c29c0, 0x17, 0x40024b1f20, 0x40003b6768, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x400248aa00, 0x113075a, 0x7, 0x40003b6768, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x400248a9c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 80 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x40034d0fa0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 93 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40003007e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 72 [IO wait]: internal/poll.runtime_pollWait(0xffff741334e8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4002410818, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4002410800, 0x40024da000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4002410800, 0x40024da000, 0x1000, 0x1000, 0x4000173688, 0x69f178, 0x40001736f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000104a0, 0x40024da000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400250cd80, 0x40024da000, 0x1000, 0x1000, 0x5, 0x40034f6a00, 0x40001737c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000082a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000082a20, 0x68f90a, 0x40003b8b00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000082a20, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40001150b0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40001150b0, 0x40013ff000, 0x1000, 0x1000, 0x4000173998, 0x44e884, 0x4000173948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40003c4940, 0x40013ff000, 0x1000, 0x1000, 0x40003caf58, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40003c4940, 0x40013ff000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40003c4980, 0x40013ff000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000173d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000173d80, 0xa, 0x40013ff201, 0x1, 0xffff741618e0, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000173d80, 0x40013ff20a, 0x1, 0x4002391520, 0x0, 0x40013ff259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000173d80, 0x40023da60a, 0x4002391520, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400248aa00, 0x40023d8480, 0x1f, 0x4000173ee0, 0x40003b6a50, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x400248aa00, 0x113075a, 0x7, 0x40044c2610, 0xd, 0x40003b6a50, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x400248a9c0, 0x40044c2610, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 97 [IO wait]: internal/poll.runtime_pollWait(0xffff74133a58, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000362b18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000362b00, 0x40003d5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000362b00, 0x40003d5000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40003d8178, 0x40003d5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000cd80, 0x40003d5000, 0x1000, 0x1000, 0x4001397500, 0x200000003, 0x4001397500) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003ca240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003ca240, 0x40013bc00a, 0x40024af8c8, 0x4d3b10, 0x40000820c0, 0x40013bc000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003ca240, 0x0, 0x1, 0x0, 0x40000820d8, 0x40013bc000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000097080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000097080, 0x40013fc000, 0x1000, 0x1000, 0x40024af9b8, 0x0, 0x40024af988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40003c46c0, 0x40013fc000, 0x1000, 0x1000, 0x4001396780, 0x3, 0x40024afbe0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40003c46c0, 0x40013fc000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40003c4700, 0x40013fc000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40024afdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40024afdd0, 0xa, 0x40013fc201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40024afdd0, 0x40013fc20a, 0x1, 0x400244ecab, 0x0, 0x40013fc263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40024afdd0, 0x40003c830a, 0x400244ecab, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400248aa00, 0x40003be4b0, 0x24, 0x40024aff20, 0x40003b6750, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x400248aa00, 0x113075a, 0x7, 0x40003b6750, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x400248a9c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 79 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x40034d0fa0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 100 [select]: net/http.(*persistConn).readLoop(0x400250cb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 101 [select]: net/http.(*persistConn).writeLoop(0x400250cb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 104 [select]: net/http.(*persistConn).readLoop(0x400250cc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 105 [select]: net/http.(*persistConn).writeLoop(0x400250cc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 73 [select]: net/http.(*persistConn).readLoop(0x400250cd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 74 [select]: net/http.(*persistConn).writeLoop(0x400250cd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 108 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000302ac0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 31 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40003c5180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 29 [select]: net/http.(*persistConn).readLoop(0x400250d680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 30 [select]: net/http.(*persistConn).writeLoop(0x400250d680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 128 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004e42420, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 127 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004e42420, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 126 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4000121dc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 111 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000120ec0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 112 [IO wait]: internal/poll.runtime_pollWait(0xffff74133888, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4002411198, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4002411180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4002411180, 0x12d9d48, 0x40003c4780, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c656e0, 0x0, 0x40003ecae0, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c656e0, 0x400244ea00, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x40003c5e80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 113 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x4000303f80, 0x40003c6fc0, 0x40003c7030, 0x4004b00830, 0x12e8f90, 0x40013d4b60, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x4004c01934, 0x5, 0x40003c5e80, 0x12eca70, 0x40003c5d40, 0x130d478, 0x40003008c0, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40000e8620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40000e8620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 130 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40013d4b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 131 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40013d4b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 132 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x400236ccf0, 0x4004b04840) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 133 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x4000302a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 134 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x40034d1040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 135 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x40034d1040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 136 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x40000108c8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 137 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x40034d1040, 0x4004b048a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 138 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000196450, 0x1144559, 0x1b, 0x40046ffe38, 0x4004b049c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004b01350, 0x4004b049c0, 0xf1f700, 0x4000197cd0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40046fff80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004670ea0, 0x4004b01350) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 139 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 140 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400008d090, 0xd00007, 0x4004b314b8, 0x5, 0x4004b01360, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 141 [IO wait]: internal/poll.runtime_pollWait(0xffff74133400, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1998, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f1980, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f1980, 0x0, 0x4000336d88, 0x665fe4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004b47920, 0x611a6cbb, 0x4000336db8, 0x445c24) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004b47920, 0x4000336e20, 0x4000336e28, 0x18, 0x40003f9500) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x4004454c40, 0x12ed9d0, 0x4004b47920, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x4004454c40, 0x12ed9d0, 0x4004b47920, 0x4000158380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 142 [IO wait]: internal/poll.runtime_pollWait(0xffff74133318, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1a98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f1a80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f1a80, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004b47938, 0xffff9c1c05b8, 0xffff74179298, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004b47938, 0x665e1c, 0x0, 0xfe5e60, 0x40025775c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004b47950, 0x4004ae09f0, 0x11ae500, 0x4004454da0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x4004454d20, 0x12ec710, 0x4004b47950, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x4004454d20, 0x12ec710, 0x4004b47950, 0x4000158380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 155 [IO wait]: internal/poll.runtime_pollWait(0xffff74039ea8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004e4e098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004e4e080, 0x4004e01000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004e4e080, 0x4004e01000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000128428, 0x4004e01000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400013b8c0, 0x4004e01000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004aef5c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004aef5c0, 0xa, 0x0, 0x0, 0x0, 0x0, 0x4002578438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004aef5c0, 0x0, 0x4002578478, 0x655784, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004d97650) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004d97650, 0x4004b07a00, 0x200, 0x200, 0x4, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004b76940, 0x4004b07a00, 0x200, 0x200, 0x4004679247, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004b76940, 0x4004b07a00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004b76980, 0x4004b07a00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004d9bb80, 0x0, 0x40025786b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004d9bb80, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004d9bb80, 0xee0d40, 0x400008d860, 0x12db068, 0x4004b76980) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004dcc480, 0x4004ddc180, 0x4004b05c20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 123 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40013d5260) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 117 [IO wait]: internal/poll.runtime_pollWait(0xffff74039dc0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400012db98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400012db80, 0x4004ae7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400012db80, 0x4004ae7000, 0x1000, 0x1000, 0x400238bbe0, 0x469b10, 0x40003f9b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001282f8, 0x4004ae7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004b7e6c0, 0x4004ae7000, 0x1000, 0x1000, 0x400238bcf8, 0x67867c, 0x40046b5d40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004aee480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004aee480, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004b7e6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 148 [IO wait]: internal/poll.runtime_pollWait(0xffff741335d0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4002410718, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4002410700, 0x40013fd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4002410700, 0x40013fd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40003d8198, 0x40013fd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400250cc60, 0x40013fd000, 0x1000, 0x1000, 0x0, 0x0, 0x4004d91b28) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003cb7a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003cb7a0, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003cb7a0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004d96b70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004d96b70, 0x4004b06a00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004b76440, 0x4004b06a00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004b76440, 0x4004b06a00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004b76480, 0x4004b06a00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004d9a420, 0x0, 0x4002577eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004d9a420, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004d9a420, 0xee0d40, 0x400008d590, 0x12db068, 0x4004b76480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004673cb0, 0x40046716e0, 0x4004b04e40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 118 [select]: net/http.(*persistConn).writeLoop(0x4004b7e6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 122 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40013d5260) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 121 [semacquire]: sync.runtime_Semacquire(0x4004ddfc88) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004ddfc80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x40000d2680, 0x4000097260, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffdc054969, 0x29, 0xffffdc0549cb, 0x25, 0x4004e864e0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004c7e620, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffdc054969, 0x29, 0xffffdc0549cb, 0x25, 0x4004e864e0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004c7e620, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004edb190, 0x4004edae60, 0x4000097260, 0x400036f680, 0x40003b0540, 0x0, 0x17f, 0x40046fcda8, 0xffff6ca03be8, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x4000158380, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004c7e620, 0xb, 0x4004c7e618, 0x6, 0x4004c7e630, 0xd, 0x4004c666e0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x4000158380, 0x4000320000, 0x4000114840) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x4000158380, 0x40003de610, 0x2, 0x2, 0x40003de620, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 129 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4004e42420) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 162 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40013d5260) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 163 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000196450, 0x1144559, 0x1b, 0x4004db3e38, 0x4004bb2ae0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004ba0f60, 0x4004bb2ae0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004db3f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004aeeae0, 0x4004ba0f60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 168 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).processSchedIndexes(0x4004e422c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:296 +0x44 created by github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:218 +0x3c goroutine 166 [select]: net/http.(*persistConn).readLoop(0x400013b8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 167 [select]: net/http.(*persistConn).writeLoop(0x400013b8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 156 [runnable]: net/http.(*persistConn).roundTrip(0x400250d680, 0x4004e5a240, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004b9df00, 0xf8, 0x10a8660, 0x400016ce18) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004b9df00, 0x1b6e900, 0xc03eb92cca4a2d87, 0x1bfb17af65) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004b9de00, 0x12db048, 0x1b6e900, 0xc03eb92cca4a2d87, 0x1bfb17af65, 0x1be4a80, 0x0, 0x4000128488, 0x4004e58600, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004e585d0, 0x4004b9de00, 0xc03eb92cca4a2d87, 0x1bfb17af65, 0x1be4a80, 0x4000128488, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004e585d0, 0x4004b9de00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004be2ae0, 0x1b, 0x400016d580, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004e093b0, 0x112f905, 0x6, 0x12db908, 0x4000047340, 0xee40c0, 0x4004e4e218, 0x400016d580, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x40034ce400, 0x46, 0x12db908, 0x4000047340, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x40034ce400, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x20, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400016dd40, 0x4, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x4004e46000, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x4004e46000, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000121e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1267 +0xf8 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 157 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x4004e422c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:224 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 158 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x4004e422c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:612 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 159 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x4004e422c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:662 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 160 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x4004e422c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:752 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 161 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004675e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:249 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 198 [IO wait]: internal/poll.runtime_pollWait(0xffff74039a20, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004e4e398, 0x72, 0x0, 0x1, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004e4e380, 0x4004e588e1, 0x1, 0x1, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004e4e380, 0x4004e588e1, 0x1, 0x1, 0x40013f5701, 0x0, 0x1c1c358) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001284a8, 0x4004e588e1, 0x1, 0x1, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*connReader).backgroundRead(0x4004e588d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:692 +0x50 created by net/http.(*connReader).startBackgroundRead /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:688 +0xc4 goroutine 179 [select]: net/http.(*persistConn).roundTrip(0x400000c6c0, 0x4000081680, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x40000fd000, 0xf8, 0x10a8660, 0x40024acdd8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x40000fd000, 0x1b6e900, 0xc03eb92cca4ddfe6, 0x1bfb1b61c4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x40000fcf00, 0x12db048, 0x1b6e900, 0xc03eb92cca4ddfe6, 0x1bfb1b61c4, 0x1be4a80, 0x0, 0x40003d8520, 0x4004c316e0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004c316b0, 0x40000fcf00, 0xc03eb92cca4ddfe6, 0x1bfb1b61c4, 0x1be4a80, 0x40003d8520, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004c316b0, 0x40000fcf00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004c146c0, 0x1b, 0x40024ad540, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004cf6480, 0x112f905, 0x6, 0x12db908, 0x40003c9340, 0xee40c0, 0x4002411398, 0x40024ad540, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x40024aa280, 0x46, 0x12db908, 0x40003c9340, 0x2, 0x4, 0x40023cdba0, 0x2, 0x4, 0x40023cdbc0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x40024aa280, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x240, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40024add00, 0x4004c315f0, 0xffff740e01f8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x40034cafc0, 0xd47600, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x40034cafc0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x4000158380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9702 +0x344 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 180 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x4000158380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1090 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 181 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x4000158380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1017 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 174 [IO wait]: internal/poll.runtime_pollWait(0xffff74039cd8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4002411498, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4002411480, 0x4004f17000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4002411480, 0x4004f17000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001284b8, 0x4004f17000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c6c0, 0x4004f17000, 0x1000, 0x1000, 0x60, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004f10240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004f10240, 0x1, 0x4004f0e360, 0x43d584, 0x0, 0x0, 0x4002573568) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000c6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 169 [IO wait]: internal/poll.runtime_pollWait(0xffff74039bf0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000362518, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000362500, 0x4004f16000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000362500, 0x4004f16000, 0x1000, 0x1000, 0x0, 0x4004f13000, 0x4002497a38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001284b0, 0x4004f16000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).fill(0x4004f10180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004f10180, 0x40a, 0x4002497aa0, 0x0, 0xffff00001001, 0x400, 0x400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).ReadLine(0x4004f10180, 0x1000, 0xffff74036c60, 0x1, 0x4002497bc8, 0x412ff8, 0x4004e098c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:389 +0x30 net/textproto.(*Reader).readLineSlice(0x4002497cf0, 0x4004e098c0, 0x4002497c48, 0xffff74036c60, 0x2c8, 0xffff9c1c05e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/textproto/reader.go:57 +0xac net/textproto.(*Reader).ReadLine(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/textproto/reader.go:38 net/http.ReadResponse(0x4004f10180, 0x4004f0c300, 0x1000, 0x4004f16000, 0x40001284b0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/response.go:161 +0x8c github.com/couchbase/cbauth/revrpc.(*Service).Run(0x40025630b0, 0x4002497f10, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:132 +0x2ac github.com/couchbase/cbauth/revrpc.BabysitService(0x4002497f10, 0x40025630b0, 0x0, 0x0, 0x0, 0xd8d03c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth/service.RegisterManager(0x13007d8, 0x4004675e00, 0x0, 0x0, 0x4000069f80, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:146 +0xa0 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).registerWithServer(0x4004675e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:301 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:205 +0x78 goroutine 170 [select]: net/http.(*persistConn).roundTrip(0x400013bd40, 0x4004e5a280, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004f0c100, 0xf8, 0x10a8660, 0x4004d8f798) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004f0c100, 0x1b6e900, 0xc03eba3aca4f8416, 0x117702135f4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004f0c000, 0x12db048, 0x1b6e900, 0xc03eba3aca4f8416, 0x117702135f4, 0x1be4a80, 0x0, 0x4000128498, 0x4004e586f0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004e586c0, 0x4004f0c000, 0xc03eba3aca4f8416, 0x117702135f4, 0x1be4a80, 0x4000128498, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004e586c0, 0x4004f0c000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004e7a940, 0x1f, 0x4004d8fd10, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/indexer.getWithAuth(0x4004e7a940, 0x1f, 0xe, 0x113846e, 0x11) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:3077 +0x68 github.com/couchbase/indexing/secondary/indexer.getGlobalTopology(0x4004ddf2a0, 0xe, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:2266 +0x78 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).updateNodeList(0x4004675e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:220 +0x3c created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:206 +0x98 goroutine 196 [IO wait]: internal/poll.runtime_pollWait(0xffff74039b08, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004e4e318, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004e4e300, 0x4004d1e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004e4e300, 0x4004d1e000, 0x1000, 0x1000, 0x40025783e8, 0x7ff1a8, 0x40025784b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40003d8530, 0x4004d1e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400013bd40, 0x4004d1e000, 0x1000, 0x1000, 0x60, 0x0, 0x4004e7a901) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004d1c060) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004d1c060, 0x1, 0x4004c74a20, 0x43d584, 0x0, 0xd8c91c, 0x4002578568) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400013bd40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 173 [select]: net/http.(*persistConn).roundTrip(0x400000cea0, 0x4000081940, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x40000fd200, 0xf8, 0x10a8660, 0x4004db0588) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x40000fd200, 0x1b6e900, 0xc03eb92cca55dce2, 0x1bfb235ec0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x40000fd100, 0x12db048, 0x1b6e900, 0xc03eb92cca55dce2, 0x1bfb235ec0, 0x1be4a80, 0x0, 0x40003d8540, 0x4004c31aa0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004c31a70, 0x40000fd100, 0xc03eb92cca55dce2, 0x1bfb235ec0, 0x1be4a80, 0x40003d8540, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004c31a70, 0x40000fd100, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004c14810, 0x1b, 0x4004db0cf0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004cf6900, 0x112f905, 0x6, 0x12db908, 0x40003c9380, 0xee40c0, 0x4002411518, 0x4004db0cf0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x40024aa300, 0x46, 0x12db908, 0x40003c9380, 0xffff6caf2f00, 0xffff6cc89fff, 0x4004db0d98, 0x41cff4, 0xffff6caf2e00, 0x100100000000, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x40024aa300, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x4004c319b0, 0xffff740e01f8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004d154b0, 0x1063520, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x40034cb200, 0x40034cb200, 0x40024aa300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.FetchNewClusterInfoCache(0xffffdc05494e, 0xe, 0x113075a, 0x7, 0x1130e44, 0x8, 0x4004e5f0c4, 0x2, 0xf01) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:123 +0x134 github.com/couchbase/indexing/secondary/manager.(*IndexManager).FetchNewClusterInfoCache(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:306 github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).getIndexMetadata(0x1be5520, 0x12ec7a0, 0x4000081900, 0x40003f0dc0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:1284 +0x74 github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).handleIndexMetadataRequest(0x1be5520, 0x12edbe0, 0x40000e8700, 0x4004f0c200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:1271 +0x3e8 net/http.HandlerFunc.ServeHTTP(0x4004b01250, 0x12edbe0, 0x40000e8700, 0x4004f0c200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2069 +0x40 net/http.(*ServeMux).ServeHTTP(0x4004b76280, 0x12edbe0, 0x40000e8700, 0x4004f0c200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2448 +0x190 net/http.serverHandler.ServeHTTP(0x4004454c40, 0x12edbe0, 0x40000e8700, 0x4004f0c200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2887 +0xbc net/http.(*conn).serve(0x4004b9eb40, 0x12f86a0, 0x4004e5a2c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:1952 +0x71c created by net/http.(*Server).Serve /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:3013 +0x318 goroutine 175 [select]: net/http.(*persistConn).writeLoop(0x400000c6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 197 [select]: net/http.(*persistConn).writeLoop(0x400013bd40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 201 [IO wait]: internal/poll.runtime_pollWait(0xffff74039938, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4002411618, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4002411600, 0x4004d24000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4002411600, 0x4004d24000, 0x1000, 0x1000, 0x40025783e8, 0x7ff1a8, 0x40025784b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40003d8550, 0x4004d24000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000cea0, 0x4004d24000, 0x1000, 0x1000, 0x60, 0x0, 0x4004e7a901) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004d1c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004d1c5a0, 0x1, 0x4004c74e40, 0x43d584, 0x0, 0xd8c91c, 0x4002578568) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000cea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 202 [select]: net/http.(*persistConn).writeLoop(0x400000cea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 182 [sleep]: time.Sleep(0x12a05f200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/plasma.RunMemQuotaTunerIteration(0x4000069fb0, 0xdf74d8, 0x400016a540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:211 +0x24c github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:247 +0xc0 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:48:48.328+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:48:48.328+00:00 [Info] Setting ipv6=false 2021-08-16T13:48:48.333+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:48:48.339+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:48:48.339+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:48:48.339+00:00 [Info] Setting log level to Info 2021-08-16T13:48:48.339+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:48:48.342+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:48:48 audit: created new audit service 2021-08-16T13:48:48.350+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:48:48.350+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:48:48.350+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:48:48.350+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:48:48.350+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:48:48.361+00:00 [Info] security context initialized 2021-08-16T13:48:48.361+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:48:48.362+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:48:48.362+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:48:48.362+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:48:48.362+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:48:48.375+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:48:48.376+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:48:48.394+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:48:48.406+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:48:48.406+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:48:48.407+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:48:48.408+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:48:48.410+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:48:48.410+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:48:48.410+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:48:48.410+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:48:48.459+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:48:48.459+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 88 us 2021-08-16T13:48:48.459+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:48:48.461+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:48:48.462+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:48:48.462+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:48:48.462+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:48:48.462+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:48:48.462+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:48:48.462+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:48:48.465+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:48.465+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:48.467+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:48:48.467+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:48:48.467+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:48:48.467+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:48.467+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:48:48.467+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:48:48.467+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:48:48.467+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:48:48.468+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:48:48.468+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:48:48.468+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:48:48.473+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 5.692463ms 2021-08-16T13:48:48.474+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:48:48.475+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:48:48.475+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:48.475+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:48:48.475+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:48:48.475+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:48:48.475+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:48:48.485+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:48:48.485+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:48:48.485+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:48:48.485+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:48:48.486+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x40000fcd9f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x40000fcd9f pc=0xbd8ea4] goroutine 187 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004dc09f0 sp=0x4004dc09c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004dc0a30 sp=0x4004dc09f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x4004cf8340, 0x4004e9e000, 0x11a9890, 0x4004dc0b40, 0x4004cf8358, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004dc0ab0 sp=0x4004dc0a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x4004cf8340, 0x4004e9e000, 0x11a9890, 0x4004dc0b40, 0x4004cf8358, 0xffffa9f28a90, 0x4004dc0b78, 0x412ddc) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004dc0af0 sp=0x4004dc0ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x40000beb40, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004dc0da0 sp=0x4004dc0af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x40000beb40, 0x40004af2d0, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004dc0e60 sp=0x4004dc0da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004cf6560, 0x4004cf6570, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004dc1200 sp=0x4004dc0e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004cf6560, 0x4004cf6570, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004dc18f0 sp=0x4004dc1200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004cabd10, 0x4004e84380, 0x40000e1b80, 0x4004cf6540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004dc1fb0 sp=0x4004dc18f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004dc1fb0 sp=0x4004dc1fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x400048c000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x40000a1890, 0x4, 0x1136a87, 0xf, 0x4000199e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 20 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x40000b2108) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 21 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x40000b2120) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 22 [IO wait]: internal/poll.runtime_pollWait(0xffff80254758, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000190118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000190100, 0x40001a1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000190100, 0x40001a1000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001364f0, 0x40001a1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x4000138480, 0x40001ac001, 0xdff, 0xdff, 0x400011b8e8, 0x6902d8, 0x4000195948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x40001285a0, 0x40001ac001, 0xdff, 0xdff, 0x40001959f8, 0x6f1968, 0x4000195938) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x400020e000, 0xa, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400020e000, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400020e000, 0xf5a960, 0x40002000e0, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x40002000c0, 0x40001482c0, 0x4000195b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x4000206000, 0x12f8f98, 0x40002000c0, 0x4000132be7, 0x16, 0x400014f320, 0x400011bc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x4000206000, 0x12f8f98, 0x40002000c0, 0x4000206000, 0x40000360c0, 0x40000360d0, 0x400020c000, 0x4000046060, 0xee0980, 0x400014f320, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x4000206000, 0x12f8f98, 0x40002000c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x40000a0810, 0x400011bf78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400011df78, 0x40000a0810, 0x12db8a8, 0x4000128570, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x40000a0810, 0x40000bc3c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x40000a0810, 0x40000bc3c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 23 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 24 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 25 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 26 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 27 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 36 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400010e300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 28 [syscall]: syscall.Syscall(0x3f, 0x0, 0x400013a7a0, 0x4, 0x400013a668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x400013a7a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000138000, 0x400013a7a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000136000, 0x400013a7a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 41 [IO wait]: internal/poll.runtime_pollWait(0xffff80254670, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400020c118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400020c100, 0x40001ad000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400020c100, 0x40001ad000, 0x1000, 0x1000, 0x4000228a88, 0x68f7a4, 0x40001c4050) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010008, 0x40001ad000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c120, 0x40001ad000, 0x1000, 0x1000, 0xf12180, 0x400009c220, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40002003c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40002003c0, 0xa, 0x1008880, 0x400009c1e0, 0x199, 0x1000, 0x4000228b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40002003c0, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400043c690) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400043c690, 0x40001d2000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000098000, 0x40001d2000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000098000, 0x40001d2000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000098040, 0x40001d2000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40001c4000, 0x0, 0x4000228eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40001c4000, 0x0, 0x0, 0x4000228f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40001c4000, 0xee0d40, 0x400009c1e0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40000d0000, 0x40003fed80, 0x40000ce180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 85 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x400020c500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 31 [select]: net/http.(*persistConn).readLoop(0x400000c120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 32 [select]: net/http.(*persistConn).writeLoop(0x400000c120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 3 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40000a27b0, 0x12c18bc, 0x1, 0x4000119e58, 0x40000ceb40, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x4000057ec8, 0x40000ceb40, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000119fa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4000099340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 4 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4000099340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 9 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40014e4300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 7 [IO wait]: internal/poll.runtime_pollWait(0xffff80254588, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000190098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000190080, 0x400022e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000190080, 0x400022e000, 0x1000, 0x1000, 0x400022cbe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001363d8, 0x400022e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400014e6c0, 0x400022e000, 0x1000, 0x1000, 0x400022ccf8, 0x67867c, 0x4000100540) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000138e40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000138e40, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400014e6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 8 [select]: net/http.(*persistConn).writeLoop(0x400014e6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 46 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000161f00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 84 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x400007f410) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 83 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x4000201320) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 82 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40014e4000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 48 [select]: net/http.(*persistConn).readLoop(0x400000c240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 51 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x40014ba000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 59 [IO wait]: internal/poll.runtime_pollWait(0xffff80253f30, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400020c218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400020c200, 0x400252d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400020c200, 0x400252d000, 0x1000, 0x1000, 0x400444d688, 0x69f178, 0x400444d6f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000a4480, 0x400252d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c240, 0x400252d000, 0x1000, 0x1000, 0x5, 0x4002444000, 0x400444d7c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40025281e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40025281e0, 0x68f90a, 0x400444d8a8, 0x412bc0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40025281e0, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400043d800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400043d800, 0x4004456000, 0x1000, 0x1000, 0x400444d998, 0x44e884, 0x80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40014e4480, 0x4004456000, 0x1000, 0x1000, 0x40002017f8, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40014e4480, 0x4004456000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40014e44c0, 0x4004456000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400444dd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400444dd80, 0xa, 0x4004456201, 0x1, 0xffff80ca1f70, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400444dd80, 0x400445620a, 0x1, 0x4001495580, 0x0, 0x4004456259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400444dd80, 0x4002504d0a, 0x4001495580, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40004bb150, 0x4004438080, 0x1f, 0x400444dee0, 0x400443a2a0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x40004bb150, 0x113075a, 0x7, 0x40024c3d60, 0xd, 0x400443a2a0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x40004bb110, 0x40024c3d60, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 125 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40001ec620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 54 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x40014ba000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 55 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x40014ba000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 56 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x40014ba000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 102 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x40001d0fa0, 0x40045ccde0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 124 [semacquire]: sync.runtime_Semacquire(0x4004cabd18) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004cabd10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x40000e1b80, 0x40001b4510, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffc3357969, 0x29, 0xffffc33579cb, 0x25, 0x40045cdce0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004c58b30, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffc3357969, 0x29, 0xffffc33579cb, 0x25, 0x40045cdce0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004c58b30, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004dc1190, 0x4004dc0e60, 0x40001b4510, 0x40001926c0, 0x4000161f00, 0x0, 0xffff80c9efa0, 0x78, 0xffff7152fff0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x400048c000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004c58b30, 0xb, 0x4004c58b28, 0x6, 0x4004c58b40, 0xd, 0x4004c37380, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x400048c000, 0x40003f0000, 0x400043c8a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x400048c000, 0x400006d280, 0x2, 0x2, 0x400006d290, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 69 [select]: net/http.(*persistConn).readLoop(0x400000c480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 70 [select]: net/http.(*persistConn).writeLoop(0x400000c480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 49 [select]: net/http.(*persistConn).writeLoop(0x400000c240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 86 [IO wait]: internal/poll.runtime_pollWait(0xffff80254018, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400020c618, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400020c600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400020c600, 0x0, 0x400013c698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4002540a50, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4002540a50, 0x12ed9d0, 0x4002540a50, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x400020c580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 87 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40004bb040, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 88 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40004bb040, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 89 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x40004bb040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 90 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x400007fef0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 91 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x4000180150) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 92 [IO wait]: internal/poll.runtime_pollWait(0xffff80253c78, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400020c798, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400020c780, 0x4002574000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400020c780, 0x4002574000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010168, 0x4002574000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000450480, 0x4002574000, 0x1000, 0x1000, 0x4000001c80, 0x200000003, 0x4000001c80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000201c20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000201c20, 0x400023200a, 0x40024d58c8, 0x4d3b10, 0x4000138060, 0x4000232000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000201c20, 0x0, 0x1, 0xffffa9f28f40, 0x4000138078, 0x4000232000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400249cc00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400249cc00, 0x40044f9000, 0x1000, 0x1000, 0x40024d59b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40024a6500, 0x40044f9000, 0x1000, 0x1000, 0x40002017f8, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40024a6500, 0x40044f9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40024a6540, 0x40044f9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40024d5dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40024d5dd0, 0xa, 0x40044f9201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40024d5dd0, 0x40044f920a, 0x1, 0x40000aa6ee, 0x0, 0x40044f92e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40024d5dd0, 0x4000114d0a, 0x40000aa6ee, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40004bb150, 0x40000ae900, 0x17, 0x40024d5f20, 0x4002540c60, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x40004bb150, 0x113075a, 0x7, 0x4002540c60, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x40004bb110) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 93 [IO wait]: internal/poll.runtime_pollWait(0xffff80253d60, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400020c318, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400020c300, 0x40004bc000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400020c300, 0x40004bc000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010068, 0x40004bc000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c480, 0x40004bc000, 0x1000, 0x1000, 0x4000001e00, 0x200000003, 0x4000001e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000200a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000200a20, 0x400023200a, 0x400011b838, 0x43d50c, 0x400011b868, 0x44ef6c, 0x400011b850) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000200a20, 0x41c330, 0xffff80c9f798, 0x0, 0x400011b8d8, 0x41cd4c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400249cc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400249cc60, 0x400016e000, 0x1000, 0x1000, 0x400011b9b8, 0x0, 0x400011b988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40024a6580, 0x400016e000, 0x1000, 0x1000, 0x400008aa80, 0x3, 0x400011bbe0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40024a6580, 0x400016e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40024a65c0, 0x400016e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400011bdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400011bdd0, 0xa, 0x400016e201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400011bdd0, 0x400016e20a, 0x1, 0x40000aa6eb, 0x0, 0x400016e263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400011bdd0, 0x40014ee10a, 0x40000aa6eb, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40004bb150, 0x40000a6fc0, 0x24, 0x400011bf20, 0x4002540c48, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x40004bb150, 0x113075a, 0x7, 0x4002540c48, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x40004bb110) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 64 [IO wait]: internal/poll.runtime_pollWait(0xffff80253b90, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000ad098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000ad080, 0x40001a6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000ad080, 0x40001a6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000136570, 0x40001a6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40000ff440, 0x40001a6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000139ec0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000139ec0, 0x2d2c305b2c5d310a, 0x312d2c305b2c5d31, 0x5d312d2c305b2c5d, 0x2c5d312d2c305b2c, 0x5b2c5d312d2c305b, 0x305b2c5d312d2c30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000139ec0, 0x7356e4, 0xffffc335794e, 0xe, 0x2c5d312d2c305b2c, 0x5b2c5d312d2c305b) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40047ee0c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40047ee0c0, 0x40001bae00, 0x200, 0x200, 0xffffc335794e, 0xe, 0x4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40014e56c0, 0x40001bae00, 0x200, 0x200, 0x5b2c5d312d2c305b, 0x305b2c5d312d2c30, 0x2c305b2c5d312d2c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40014e56c0, 0x40001bae00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40014e5700, 0x40001bae00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40046086e0, 0x305b2c5d312d2c00, 0x40001976b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40046086e0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40046086e0, 0xee0d40, 0x400014a910, 0x12db068, 0x40014e5700) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40047ec090, 0x400447f2c0, 0x400471daa0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 96 [select]: net/http.(*persistConn).readLoop(0x4000450480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 97 [select]: net/http.(*persistConn).writeLoop(0x4000450480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 77 [select]: net/http.(*persistConn).readLoop(0x40000ff440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 78 [select]: net/http.(*persistConn).writeLoop(0x40000ff440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 163 [select]: net/http.(*persistConn).writeLoop(0x400014eb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 10 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000099300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 11 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000099bc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 12 [IO wait]: internal/poll.runtime_pollWait(0xffff802539c0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000191598, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000191580, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000191580, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c391e8, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c391e8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x4000161c00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 13 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x4004c7e0c0, 0x40000971f0, 0x4000097260, 0x4004c42850, 0x12e8f90, 0x40004f09a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x40046efe44, 0x5, 0x4000161c00, 0x12eca70, 0x4000161ac0, 0x130d478, 0x40003dc5b0, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40004f0e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40004f0e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 14 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40004f09a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 15 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40004f09a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 16 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x40014ec7e0, 0x40046d51a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 130 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x40014e42c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 131 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x40001d0fa0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 132 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x40001d0fa0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 133 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x40000a46a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 134 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x40001d1040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 135 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x40001d1040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 136 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x4004c7e0c0, 0x4004c42850) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 103 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40000a27b0, 0x1144559, 0x1b, 0x4004447e38, 0x40045ccf00, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004589a70, 0x40045ccf00, 0x305b2c5d312d2c30, 0x2c305b2c5d312d2c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x2c305b2c5d312d2c, 0x2d2c305b2c5d312d) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004447f80, 0x2d2c305b2c5d312d, 0x312d2c305b2c5d31) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004c9e4e0, 0x4004589a70) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 104 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 105 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x4000206cd0, 0xd00007, 0x40045c1ee8, 0x5, 0x4004589a80, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 106 [IO wait]: internal/poll.runtime_pollWait(0xffff80253aa8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400020d018, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400020d000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400020d000, 0xffffa9f285b8, 0x30, 0x400249df20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40045fd938, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40045fd938, 0x400249df20, 0x11ae500, 0x400021e5c0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x400021e540, 0x12ed9d0, 0x40045fd938, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x400021e540, 0x12ed9d0, 0x40045fd938, 0x400048c000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 107 [IO wait]: internal/poll.runtime_pollWait(0xffff802538d8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400020d118, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400020d100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400020d100, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40045fd950, 0xffffa9f285b8, 0xffffa9f31fe8, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40045fd950, 0x665e1c, 0x0, 0xfe5e60, 0x4000141dc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x40045fd968, 0x400249dfb0, 0x11ae500, 0x400021e6a0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x400021e620, 0x12ec710, 0x40045fd968, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x400021e620, 0x12ec710, 0x40045fd968, 0x400048c000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 111 [IO wait]: internal/poll.runtime_pollWait(0xffff80253e48, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000ad998, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000ad980, 0x4004e40000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000ad980, 0x4004e40000, 0x1000, 0x1000, 0x400461fad8, 0x412bc0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40014bc250, 0x4004e40000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400014eb40, 0x4004e40000, 0x1000, 0x1000, 0x4004c9e900, 0x60, 0x58) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400447f980) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400447f980, 0x100a, 0x312d2c305b2c5d31, 0x5d312d2c305b2c5d, 0x2c5d312d2c305b2c, 0x5b2c5d312d2c305b, 0x4000197438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400447f980, 0x7356e4, 0x4000197478, 0x655784, 0x40001974a8, 0x40b2f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40045cb440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40045cb440, 0x40045d3200, 0x200, 0x200, 0x60, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000209c80, 0x40045d3200, 0x200, 0x200, 0x40045cd080, 0x33, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000209c80, 0x40045d3200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000209cc0, 0x40045d3200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40045dfa20, 0x680d00, 0x40001976b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40045dfa20, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40045dfa20, 0xee0d40, 0x4000206eb0, 0x12db068, 0x4000209cc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400458f8c0, 0x4004c9ea20, 0x40045cd440) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 108 [IO wait]: internal/poll.runtime_pollWait(0xffff715895a8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000ad718, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000ad700, 0x4004ca8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000ad700, 0x4004ca8000, 0x1000, 0x1000, 0x4000069be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010298, 0x4004ca8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40000ffd40, 0x4004ca8000, 0x1000, 0x1000, 0x4000069cf8, 0x67867c, 0x40045cd080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004c9e7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004c9e7e0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40000ffd40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 115 [IO wait]: internal/poll.runtime_pollWait(0xffff715894c0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000191998, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000191980, 0x400460f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000191980, 0x400460f000, 0x1000, 0x1000, 0x4004440be0, 0x469b10, 0x40046df680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001366c0, 0x400460f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40014f99e0, 0x400460f000, 0x1000, 0x1000, 0x4004440cf8, 0x67867c, 0x40046064e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004604420) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004604420, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40014f99e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 116 [select]: net/http.(*persistConn).writeLoop(0x40014f99e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 109 [select]: net/http.(*persistConn).writeLoop(0x40000ffd40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 126 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40001ec620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 127 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40001ec620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 128 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40001ec620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 129 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x40024a71c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 146 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4000165810, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 147 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4000165810, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 148 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4000165810) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 149 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40001ec620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 150 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40000a27b0, 0x1144559, 0x1b, 0x4000117e38, 0x40046071a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004614bb0, 0x40046071a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000117f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40046050e0, 0x4004614bb0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 162 [select]: net/http.(*persistConn).readLoop(0x400014eb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 112 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40024a7200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 113 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x40001656b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 178 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x40001656b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 179 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x40001656b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 180 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x40001656b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 181 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x400458dc00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 182 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x400458dc00, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 183 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x400048c000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 184 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x400048c000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 185 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x400048c000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 186 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:48:53.638+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:48:53.638+00:00 [Info] Setting ipv6=false 2021-08-16T13:48:53.643+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:48:53.648+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:48:53.648+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:48:53.648+00:00 [Info] Setting log level to Info 2021-08-16T13:48:53.649+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:48:53.651+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:48:53 audit: created new audit service 2021-08-16T13:48:53.659+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:48:53.659+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:48:53.660+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:48:53.660+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:48:53.660+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:48:53.670+00:00 [Info] security context initialized 2021-08-16T13:48:53.670+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:48:53.670+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:48:53.670+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:48:53.670+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:48:53.670+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:48:53.683+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:48:53.686+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:48:53.700+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:48:53.712+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:48:53.712+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:48:53.713+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:48:53.713+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:48:53.716+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:48:53.717+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:48:53.717+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:48:53.718+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:48:53.767+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:48:53.768+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 60 us 2021-08-16T13:48:53.768+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:48:53.770+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:48:53.770+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:48:53.774+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:48:53.774+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:48:53.774+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:48:53.774+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:48:53.774+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:48:53.774+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:53.774+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:53.775+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:48:53.775+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:48:53.775+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:48:53.775+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:53.775+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:48:53.775+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:48:53.775+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:48:53.775+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:48:53.775+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:48:53.775+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:48:53.775+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:48:53.782+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 6.63662ms 2021-08-16T13:48:53.783+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:48:53.784+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:48:53.784+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:53.784+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:48:53.785+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:48:53.785+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:48:53.785+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:48:53.796+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:48:53.796+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:48:53.796+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:48:53.796+00:00 [Info] RebalanceMgr::initService Init 2021-08-16T13:48:53.796+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:48:53.797+00:00 [Info] ServiceMgr::registerWithServer nodeuuid 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:53.797+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x4004e1049f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x4004e1049f pc=0xbd8ea4] goroutine 192 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004e3a9f0 sp=0x4004e3a9c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004e3aa30 sp=0x4004e3a9f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x40000a9860, 0x4004482c00, 0x11a9890, 0x4004e3ab40, 0x40000a9878, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004e3aab0 sp=0x4004e3aa40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x40000a9860, 0x4004482c00, 0x11a9890, 0x4004e3ab40, 0x40000a9878, 0xc18f8c, 0x4004e3ac18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004e3aaf0 sp=0x4004e3aab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x400447c280, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004e3ada0 sp=0x4004e3aaf0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x400447c280, 0x40024d2e70, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004e3ae60 sp=0x4004e3ada0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x40044d18d0, 0x40044d18e0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004e3b200 sp=0x4004e3ae60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x40044d18d0, 0x40044d18e0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004e3b8f0 sp=0x4004e3b200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004e18d00, 0x4004e2e700, 0x4004476580, 0x40044d18b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004e3bfb0 sp=0x4004e3b8f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004e3bfb0 sp=0x4004e3bfb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x400046c000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x400008d7a0, 0x4, 0x1136a87, 0xf, 0x4000211e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 5 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400000e108) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 6 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400000e120) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 7 [IO wait]: internal/poll.runtime_pollWait(0xffff84474018, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000b2718, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000b2700, 0x40000e5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000b2700, 0x40000e5000, 0x1000, 0x1000, 0x1311860, 0x47b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000100b0, 0x40000e5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x40000828a0, 0x4000138001, 0xdff, 0xdff, 0x40001998e8, 0x6902d8, 0x400020d948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400000e168, 0x4000138001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x4000132000, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4000132000, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4000132000, 0xf5a960, 0x4000114140, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4000114120, 0x40001181a0, 0x400020db28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400012c000, 0x12f8f98, 0x4000114120, 0x40001108c7, 0x16, 0x400013c120, 0x4000199c01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400012c000, 0x12f8f98, 0x4000114120, 0x400012c000, 0x40001100d0, 0x40001100e0, 0x400011c080, 0x40001180c0, 0xee0980, 0x400013c120, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400012c000, 0x12f8f98, 0x4000114120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x400008c5a0, 0x4000199f78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x4000171f78, 0x400008c5a0, 0x12db8a8, 0x400000e138, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x400008c5a0, 0x40000a03c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x400008c5a0, 0x40000a03c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 8 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 35 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400010c100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 9 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 10 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 11 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 12 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 13 [syscall]: syscall.Syscall(0x3f, 0x0, 0x4000158fa0, 0x4, 0x4000158e68, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x4000158fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x40001ba000, 0x4000158fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x40001b8000, 0x4000158fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 39 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000116fc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 37 [select]: net/http.(*persistConn).readLoop(0x40001c6360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 38 [select]: net/http.(*persistConn).writeLoop(0x40001c6360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 20 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400006c7f0, 0x12c18bc, 0x1, 0x400019de58, 0x4000038c00, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x400021a6c8, 0x4000038c00, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400019dfa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4000081200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 21 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4000081200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 33 [IO wait]: internal/poll.runtime_pollWait(0xffff844737f0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400235c498, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400235c480, 0x40023de000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400235c480, 0x40023de000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b8548, 0x40023de000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400013cc60, 0x40023de000, 0x1000, 0x1000, 0x400042f380, 0x200000003, 0x400042f380) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000115a40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000115a40, 0x400040600a, 0x40023ed8c8, 0x4d3b10, 0x40001ba060, 0x4000406000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000115a40, 0x0, 0x1, 0x0, 0x40001ba078, 0x4000406000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400041cd20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400041cd20, 0x40023e1000, 0x1000, 0x1000, 0x40023ed9b8, 0x44d8a8, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40001d9e00, 0x40023e1000, 0x1000, 0x1000, 0x40003ed798, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40001d9e00, 0x40023e1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001d9e40, 0x40023e1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40023eddd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40023eddd0, 0xa, 0x40023e1201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40023eddd0, 0x40023e120a, 0x1, 0x4000036fdb, 0x0, 0x40023e1263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40023eddd0, 0x40001c260a, 0x4000036fdb, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40023e8860, 0x400011e8a0, 0x24, 0x40023edf20, 0x40003f7830, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x40023e8860, 0x113075a, 0x7, 0x40003f7830, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x40023e8820) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 24 [IO wait]: internal/poll.runtime_pollWait(0xffff84473e48, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011c198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011c180, 0x400022c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011c180, 0x400022c000, 0x1000, 0x1000, 0x4000014be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011a0d0, 0x400022c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400013c5a0, 0x400022c000, 0x1000, 0x1000, 0x4000014cf8, 0x67867c, 0x4000228360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003ed380) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40003ed380, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400013c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 25 [select]: net/http.(*persistConn).writeLoop(0x400013c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 66 [IO wait]: internal/poll.runtime_pollWait(0xffff84473f30, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000208118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000208100, 0x4000139000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000208100, 0x4000139000, 0x1000, 0x1000, 0x40003d2a88, 0x68f7a4, 0x40001345d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b80a8, 0x4000139000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c6360, 0x4000139000, 0x1000, 0x1000, 0xf12180, 0x4000098220, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001141e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001141e0, 0xa, 0x1008880, 0x40000981e0, 0x199, 0x1000, 0x40003d2b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40001141e0, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400008d740) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400008d740, 0x40003d4000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000126100, 0x40003d4000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000126100, 0x40003d4000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000126140, 0x40003d4000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4000134580, 0x0, 0x40003d2eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4000134580, 0x0, 0x0, 0x40003d2f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4000134580, 0xee0d40, 0x40000981e0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400001a3f0, 0x40000825a0, 0x4000154180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 130 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x40001bdd60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 72 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000116f80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 28 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000116c40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 64 [select]: net/http.(*persistConn).writeLoop(0x40001c66c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 63 [select]: net/http.(*persistConn).readLoop(0x40001c66c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 100 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001264c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 30 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x4000200a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 122 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x400447a620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 67 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x4000200a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 68 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x4000200a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 69 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x4000200a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 32 [IO wait]: internal/poll.runtime_pollWait(0xffff84473708, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011c318, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011c300, 0x40024f5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011c300, 0x40024f5000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010098, 0x40024f5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c66c0, 0x40024f5000, 0x1000, 0x1000, 0x4000186c00, 0x200000003, 0x4000186c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004398600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004398600, 0x400040600a, 0x40001998c8, 0x4d3b10, 0x40001ba060, 0x4000406000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004398600, 0x0, 0x1, 0x0, 0x40001ba078, 0x4000406000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40023936e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40023936e0, 0x400237e000, 0x1000, 0x1000, 0x40001999b8, 0x0, 0x4000199988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40001167c0, 0x400237e000, 0x1000, 0x1000, 0x400042f980, 0x3, 0x4000199be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40001167c0, 0x400237e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000116840, 0x400237e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000199dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000199dd0, 0xa, 0x400237e201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000199dd0, 0x400237e20a, 0x1, 0x4000036cd9, 0x0, 0x400237e2e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000199dd0, 0x40001fd10a, 0x4000036cd9, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40023e8860, 0x400003a540, 0x17, 0x4000199f20, 0x400001e648, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x40023e8860, 0x113075a, 0x7, 0x400001e648, 0x0, 0x471401, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x40023e8820) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 40 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x4000114c00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 41 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x400010ee40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 42 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x40000b2080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 43 [IO wait]: internal/poll.runtime_pollWait(0xffff844738d8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000b2198, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000b2180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000b2180, 0x0, 0x4000218698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40023c8de0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40023c8de0, 0x12ed9d0, 0x40023c8de0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x40000b2100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 44 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40001fcdd0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 45 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40001fcdd0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 46 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x40001fcdd0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 47 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x400010f920) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 90 [select]: net/http.(*persistConn).writeLoop(0x400013cc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 111 [select]: net/http.(*persistConn).writeLoop(0x4004748360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 82 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40001f8150) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 121 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x400447a620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 85 [select]: net/http.(*persistConn).readLoop(0x400000cb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 86 [select]: net/http.(*persistConn).writeLoop(0x400000cb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 91 [IO wait]: internal/poll.runtime_pollWait(0xffff84473538, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000b2498, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000b2480, 0x40023d8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000b2480, 0x40023d8000, 0x1000, 0x1000, 0x400019f688, 0x69f178, 0x400019f6f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b8530, 0x40023d8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000cb40, 0x40023d8000, 0x1000, 0x1000, 0x5, 0x4000444a00, 0x400019f7c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001158c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001158c0, 0x68f90a, 0x4000144e70, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40001158c0, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400041d020) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400041d020, 0x4002490000, 0x1000, 0x1000, 0x400019f998, 0x44e884, 0x400019f948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40001d9ec0, 0x4002490000, 0x1000, 0x1000, 0x40003ed978, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40001d9ec0, 0x4002490000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001d9f00, 0x4002490000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400019fd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400019fd80, 0xa, 0x4002490201, 0x1, 0xffff842e89b8, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400019fd80, 0x400249020a, 0x1, 0x40024bbf80, 0x0, 0x4002490259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400019fd80, 0x400009d30a, 0x40024bbf80, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40023e8860, 0x40023cad80, 0x1f, 0x400019fee0, 0x40023c93b0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x40023e8860, 0x113075a, 0x7, 0x4000036cf0, 0xd, 0x40023c93b0, 0x0, 0x0, 0x2) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x40023e8820, 0x4000036cf0, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 89 [select]: net/http.(*persistConn).readLoop(0x400013cc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 132 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x4004516f80, 0x4004566f90) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 119 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x400447a620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 131 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x40001bdd60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 77 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000081dc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 118 [semacquire]: sync.runtime_Semacquire(0x4004e18d08) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004e18d00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x4004476580, 0x400041cf60, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffe65a5969, 0x29, 0xffffe65a59cb, 0x25, 0x40044ddf80, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x40047cef50, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffe65a5969, 0x29, 0xffffe65a59cb, 0x25, 0x40044ddf80, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x40047cef50, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004e3b190, 0x4004e3ae60, 0x400041cf60, 0x40001cb8c0, 0x4000116c40, 0x0, 0x7f, 0x400016eda8, 0xffff842f3258, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x400046c000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x40047cef50, 0xb, 0x40047cef48, 0x6, 0x40047cef60, 0xd, 0x40047cbe60, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x400046c000, 0x40003e0000, 0x400041c7b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x400046c000, 0x4000112ae0, 0x2, 0x2, 0x4000112af0, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 103 [select]: net/http.(*persistConn).readLoop(0x400013d7a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 104 [select]: net/http.(*persistConn).writeLoop(0x400013d7a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 78 [IO wait]: internal/poll.runtime_pollWait(0xffff84473368, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000209018, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000209000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000209000, 0x58ec48, 0x400460ae40, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40047cd6f8, 0x4714f4, 0x400460ae40, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40047cd6f8, 0x4004642f00, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x4004796500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 79 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x4004516f80, 0x4000129030, 0x40001290a0, 0x4004566f90, 0x12e8f90, 0x40043dc700, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x40047ce264, 0x5, 0x4004796500, 0x12eca70, 0x40047963c0, 0x130d478, 0x40000fc620, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40043dc8c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40043dc8c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 133 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40043dc700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 134 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40043dc700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 135 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x400001aa20, 0x4004c80240) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 136 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x4000126480) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 137 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x40001bde00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 138 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x40001bde00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 139 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x40001b87e8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 140 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x40001bde00, 0x4004c802a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 141 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400006c7f0, 0x1144559, 0x1b, 0x40023efe38, 0x4004c803c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004567ab0, 0x4004c803c0, 0xf1f700, 0x4000460070) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40023eff80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004553320, 0x4004567ab0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 142 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 143 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400012d130, 0xd00007, 0x4004cb0c48, 0x5, 0x4004567ac0, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 144 [runnable]: internal/poll.runtime_pollWait(0xffff84473280, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000b3d18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000b3d00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000b3d00, 0xffffaebab5b8, 0x30, 0x4004cc8060) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004cb82b8, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004cb82b8, 0x4004cc8060, 0x11ae500, 0x4000214be0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x4000214b60, 0x12ed9d0, 0x4004cb82b8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x4000214b60, 0x12ed9d0, 0x4004cb82b8, 0x400046c000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 145 [IO wait]: internal/poll.runtime_pollWait(0xffff84473198, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000b3e18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000b3e00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000b3e00, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004cb82d0, 0xffffaebab5b8, 0xffff8425adc0, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004cb82d0, 0x665e1c, 0x0, 0xfe5e60, 0x4002480dc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004cb82e8, 0x4004cc80f0, 0x11ae500, 0x4000214cc0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x4000214c40, 0x12ec710, 0x4004cb82e8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x4000214c40, 0x12ec710, 0x4004cb82e8, 0x400046c000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 152 [IO wait]: internal/poll.runtime_pollWait(0xffff84473450, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011d918, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011d900, 0x4004e22000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011d900, 0x4004e22000, 0x1000, 0x1000, 0x1, 0x4004cc8c00, 0x4004cd0a78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000104c8, 0x4004e22000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c79e0, 0x4004e22000, 0x1000, 0x1000, 0x40045175c0, 0x0, 0x105f560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004399c80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004399c80, 0x5b2c5d312d2c300a, 0x305b2c5d312d2c30, 0x2c305b2c5d312d2c, 0x2d2c305b2c5d312d, 0x312d2c305b2c5d31, 0x5d312d2c305b2c5d) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004399c80, 0x5b2c5d312d2c305b, 0x305b2c5d312d2c30, 0x2c305b2c5d312d2c, 0x2d2c305b2c5d312d, 0x312d2c305b2c5d31) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004cc9140) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004cc9140, 0x4004c7d600, 0x200, 0x200, 0x2c5d312d2c305b2c, 0x5b2c5d312d2c305b, 0x305b2c5d312d2c30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40045177c0, 0x4004c7d600, 0x200, 0x200, 0x312d2c305b2c5d31, 0x5d312d2c305b2c5d, 0x2c5d312d2c305b2c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40045177c0, 0x4004c7d600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004517800, 0x4004c7d600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004d74160, 0x0, 0x4002487eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004d74160, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004d74160, 0xee0d40, 0x400012d900, 0x12db068, 0x4004517800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004cb7050, 0x4004d703c0, 0x4004c81260) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 112 [IO wait]: internal/poll.runtime_pollWait(0xffff84473620, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400235ca98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400235ca80, 0x400468a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400235ca80, 0x400468a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011a2f8, 0x400468a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400013d7a0, 0x400468a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002395a40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002395a40, 0x100a, 0x0, 0x0, 0x0, 0x0, 0x400015cc38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4002395a40, 0x0, 0x400015cc78, 0x655784, 0x400015cca8, 0x40b2f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40045fbec0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40045fbec0, 0x400471f200, 0x200, 0x200, 0x4, 0x0, 0x400015cd08) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000117d40, 0x400471f200, 0x200, 0x200, 0x400470cb87, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000117d40, 0x400471f200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000117d80, 0x400471f200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x400472d760, 0x680d00, 0x400015ceb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400472d760, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400472d760, 0xee0d40, 0x4000180af0, 0x12db068, 0x4000117d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004d96000, 0x400470ed80, 0x4004d923c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 106 [IO wait]: internal/poll.runtime_pollWait(0xffff6e1ee100, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400235d198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400235d180, 0x400474e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400235d180, 0x400474e000, 0x1000, 0x1000, 0x40003cfbe0, 0x469b10, 0x4004cbb200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011a400, 0x400474e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004caeb40, 0x400474e000, 0x1000, 0x1000, 0x40003cfcf8, 0x67867c, 0x40045ffe60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400470e8a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x400470e8a0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004caeb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 107 [select]: net/http.(*persistConn).writeLoop(0x4004caeb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 110 [IO wait]: internal/poll.runtime_pollWait(0xffff6e1ee1e8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400235d298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400235d280, 0x4004d94000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400235d280, 0x4004d94000, 0x1000, 0x1000, 0x4004cccbe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011a428, 0x4004d94000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004748360, 0x4004d94000, 0x1000, 0x1000, 0x4004ccccf8, 0x67867c, 0x4004d92360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400470ecc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x400470ecc0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004748360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 120 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x400447a620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 123 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4000127200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 124 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40024cf760, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 125 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40024cf760, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 126 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x40024cf760) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 127 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x400447a620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 128 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400006c7f0, 0x1144559, 0x1b, 0x40023f1e38, 0x40044dd380, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x40044d1570, 0x40044dd380, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40023f1f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004399800, 0x40044d1570) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 181 [select]: net/http.(*Transport).getConn(0x1b6e900, 0x4004517b00, 0x0, 0x4004cbe9c0, 0x4, 0x4004ea4250, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1368 +0x418 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004571d00, 0xf8, 0x10a8660, 0x4004cd4e18) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:579 +0x5c4 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004571d00, 0x1b6e900, 0xc03eb92f6f7f094e, 0x1bfa4b2da6) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004571c00, 0x12db048, 0x1b6e900, 0xc03eb92f6f7f094e, 0x1bfa4b2da6, 0x1be4a80, 0x0, 0x40001b8930, 0x4004cc9800, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004cc97d0, 0x4004571c00, 0xc03eb92f6f7f094e, 0x1bfa4b2da6, 0x1be4a80, 0x40001b8930, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004cc97d0, 0x4004571c00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004cbe990, 0x1b, 0x4004cd5580, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004cb79e0, 0x112f905, 0x6, 0x12db908, 0x40001c37a0, 0xee40c0, 0x4004d6c518, 0x4004cd5580, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x400003e400, 0x46, 0x12db908, 0x40001c37a0, 0x5b2c5d312d2c305b, 0x305b2c5d312d2c30, 0x2c305b2c5d312d2c, 0x2d2c305b2c5d312d, 0x312d2c305b2c5d31, 0x5d312d2c305b2c5d, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x400003e400, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x20, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004cd5d40, 0x4, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x40000dd680, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x40000dd680, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000127240) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1267 +0xf8 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 179 [select]: net/http.(*persistConn).readLoop(0x40001c79e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 180 [select]: net/http.(*persistConn).writeLoop(0x40001c79e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 182 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x40024cf600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:224 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 183 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x40024cf600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:612 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 184 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x40024cf600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:662 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 185 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x40024cf600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:752 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 186 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004e0c400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:249 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 158 [runnable]: internal/poll.runtime_pollWait(0xffff6e1edf30, 0x77, 0x4004e0b500) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004d6c718, 0x77, 0x12f8600, 0x4004d711a0, 0x4004d6c700) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitWrite(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:96 internal/poll.(*FD).WaitWrite(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:528 net.(*netFD).connect(0x4004d6c700, 0x12f8668, 0x4004d711a0, 0x0, 0x0, 0x12db428, 0x4004e9e5e0, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:141 +0x1e4 net.(*netFD).dial(0x4004d6c700, 0x12f8668, 0x4004d711a0, 0x12ff2c0, 0x0, 0x12ff2c0, 0x4004cc9a70, 0x0, 0xffff8425a6d8, 0x10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:149 +0xac net.socket(0x12f8668, 0x4004d711a0, 0x112eab2, 0x3, 0x2, 0x1, 0x0, 0x0, 0x12ff2c0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:70 +0x148 net.internetSocket(0x12f8668, 0x4004d711a0, 0x112eab2, 0x3, 0x12ff2c0, 0x0, 0x12ff2c0, 0x4004cc9a70, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/ipsock_posix.go:141 +0xcc net.(*sysDialer).doDialTCP(0x4004d6c680, 0x12f8668, 0x4004d711a0, 0x0, 0x4004cc9a70, 0xf74f40, 0x1c1c218, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:65 +0x8c net.(*sysDialer).dialTCP(0x4004d6c680, 0x12f8668, 0x4004d711a0, 0x0, 0x4004cc9a70, 0x46e908, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:61 +0xb8 net.(*sysDialer).dialSingle(0x4004d6c680, 0x12f8668, 0x4004d711a0, 0x12e9300, 0x4004cc9a70, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:580 +0x4d4 net.(*sysDialer).dialSerial(0x4004d6c680, 0x12f8668, 0x4004d711a0, 0x4004d72950, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:548 +0xf8 net.(*Dialer).DialContext(0x40001ba240, 0x12f8668, 0x4004d711a0, 0x112eab2, 0x3, 0x4004ea42b0, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:425 +0x550 net/http.(*Transport).dial(0x1b6e900, 0x12f8668, 0x4004d70f00, 0x112eab2, 0x3, 0x4004ea42b0, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1162 +0x178 net/http.(*Transport).dialConn(0x1b6e900, 0x12f8668, 0x4004d70f00, 0x0, 0x4004cbeab0, 0x4, 0x4004ea42b0, 0xe, 0x0, 0x4004caf200, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1600 +0x155c net/http.(*Transport).dialConnFor(0x1b6e900, 0x4004c82c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1442 +0x90 created by net/http.(*Transport).queueForDial /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1411 +0x3a8 goroutine 188 [select]: net/http.(*persistConn).roundTrip(0x4004748360, 0x4004517ac0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004571b00, 0xf8, 0x10a8660, 0x400016cdd8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004571b00, 0x1b6e900, 0xc03eb92f6f7df190, 0x1bfa4a15e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004571a00, 0x12db048, 0x1b6e900, 0xc03eb92f6f7df190, 0x1bfa4a15e0, 0x1be4a80, 0x0, 0x40001b8920, 0x4004cc9680, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004cc9650, 0x4004571a00, 0xc03eb92f6f7df190, 0x1bfa4a15e0, 0x1be4a80, 0x40001b8920, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004cc9650, 0x4004571a00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004cbe870, 0x1b, 0x400016d540, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004cb7710, 0x112f905, 0x6, 0x12db908, 0x40001c3760, 0xee40c0, 0x4004d6c498, 0x400016d540, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x40024c6500, 0x46, 0x12db908, 0x40001c3760, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x40024c6500, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x240, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400016dd00, 0x4004cc0240, 0xffff84259cc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x4004ea6000, 0xd47600, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x4004ea6000, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x400046c000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9702 +0x344 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 189 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x400046c000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1090 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 190 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x400046c000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1017 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 153 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).processSchedIndexes(0x40024cf600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:296 +0x44 created by github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:218 +0x3c goroutine 191 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 goroutine 154 [runnable]: internal/poll.runtime_pollWait(0xffff6e1ee018, 0x77, 0x4004cbbe00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004d6c618, 0x77, 0x12f8600, 0x4004d70de0, 0x4004d6c600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitWrite(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:96 internal/poll.(*FD).WaitWrite(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:528 net.(*netFD).connect(0x4004d6c600, 0x12f8668, 0x4004d70de0, 0x0, 0x0, 0x12db428, 0x4004e9e5a0, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:141 +0x1e4 net.(*netFD).dial(0x4004d6c600, 0x12f8668, 0x4004d70de0, 0x12ff2c0, 0x0, 0x12ff2c0, 0x4004cc98f0, 0x0, 0xffff8425a6d8, 0x10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:149 +0xac net.socket(0x12f8668, 0x4004d70de0, 0x112eab2, 0x3, 0x2, 0x1, 0x0, 0x0, 0x12ff2c0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:70 +0x148 net.internetSocket(0x12f8668, 0x4004d70de0, 0x112eab2, 0x3, 0x12ff2c0, 0x0, 0x12ff2c0, 0x4004cc98f0, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/ipsock_posix.go:141 +0xcc net.(*sysDialer).doDialTCP(0x4004d6c580, 0x12f8668, 0x4004d70de0, 0x0, 0x4004cc98f0, 0xf74f40, 0x1c1c218, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:65 +0x8c net.(*sysDialer).dialTCP(0x4004d6c580, 0x12f8668, 0x4004d70de0, 0x0, 0x4004cc98f0, 0x46e908, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:61 +0xb8 net.(*sysDialer).dialSingle(0x4004d6c580, 0x12f8668, 0x4004d70de0, 0x12e9300, 0x4004cc98f0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:580 +0x4d4 net.(*sysDialer).dialSerial(0x4004d6c580, 0x12f8668, 0x4004d70de0, 0x4004d72840, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:548 +0xf8 net.(*Dialer).DialContext(0x40001ba240, 0x12f8668, 0x4004d70de0, 0x112eab2, 0x3, 0x4004ea4250, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:425 +0x550 net/http.(*Transport).dial(0x1b6e900, 0x12f8668, 0x4004d70ae0, 0x112eab2, 0x3, 0x4004ea4250, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1162 +0x178 net/http.(*Transport).dialConn(0x1b6e900, 0x12f8668, 0x4004d70ae0, 0x0, 0x4004cbe9c0, 0x4, 0x4004ea4250, 0xe, 0x0, 0x4004caefc0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1600 +0x155c net/http.(*Transport).dialConnFor(0x1b6e900, 0x4004c82bb0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1442 +0x90 created by net/http.(*Transport).queueForDial /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1411 +0x3a8 goroutine 155 [select]: net.(*netFD).connect.func2(0x12f8668, 0x4004d70de0, 0x4004d6c600, 0x4004c81b00, 0x4004c81aa0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:118 +0x70 created by net.(*netFD).connect /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:117 +0x1c0 goroutine 156 [runnable]: net.(*netFD).connect(0x4004d6c800, 0x12f8630, 0x40001b4058, 0x0, 0x0, 0x12db428, 0x4004e9e620, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:99 +0x58c net.(*netFD).dial(0x4004d6c800, 0x12f8630, 0x40001b4058, 0x12ff2c0, 0x0, 0x12ff2c0, 0x4004cc9b60, 0x0, 0xffff8425a6d8, 0x10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:149 +0xac net.socket(0x12f8630, 0x40001b4058, 0x112eab2, 0x3, 0x2, 0x1, 0x0, 0x0, 0x12ff2c0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:70 +0x148 net.internetSocket(0x12f8630, 0x40001b4058, 0x112eab2, 0x3, 0x12ff2c0, 0x0, 0x12ff2c0, 0x4004cc9b60, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/ipsock_posix.go:141 +0xcc net.(*sysDialer).doDialTCP(0x4004d6c780, 0x12f8630, 0x40001b4058, 0x0, 0x4004cc9b60, 0x0, 0x4004cd1978, 0x41ab8c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:65 +0x8c net.(*sysDialer).dialTCP(0x4004d6c780, 0x12f8630, 0x40001b4058, 0x0, 0x4004cc9b60, 0xf89d40, 0x4004c81e2c, 0x100) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:61 +0xb8 net.(*sysDialer).dialSingle(0x4004d6c780, 0x12f8630, 0x40001b4058, 0x12e9300, 0x4004cc9b60, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:580 +0x4d4 net.(*sysDialer).dialSerial(0x4004d6c780, 0x12f8630, 0x40001b4058, 0x4004d72990, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:548 +0xf8 net.(*Dialer).DialContext(0x4004ebfcb8, 0x12f8630, 0x40001b4058, 0x112eab2, 0x3, 0x4004c81e2c, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:425 +0x550 net.(*Dialer).Dial(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:348 net.Dial(0x112eab2, 0x3, 0x4004c81e2c, 0xe, 0x30, 0x1008940, 0x1, 0x4004cc9b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:319 +0x7c github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4004cc9b00, 0x4004ebff10, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:116 +0xc0 github.com/couchbase/cbauth/revrpc.BabysitService(0x4004ebff10, 0x4004cc9b00, 0x0, 0x0, 0x0, 0xd8d03c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth/service.RegisterManager(0x13007d8, 0x4004e0c400, 0x0, 0x0, 0x4004cd1f80, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:146 +0xa0 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).registerWithServer(0x4004e0c400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:301 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:205 +0x78 goroutine 157 [select]: net/http.(*Transport).getConn(0x1b6e900, 0x4004517b40, 0x0, 0x4004cbeab0, 0x4, 0x4004ea42b0, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1368 +0x418 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004571f00, 0xf8, 0x10a8660, 0x4004cd1798) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:579 +0x5c4 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004571f00, 0x1b6e900, 0xc03eba3d6f80ae1a, 0x1176f510272) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004571e00, 0x12db048, 0x1b6e900, 0xc03eba3d6f80ae1a, 0x1176f510272, 0x1be4a80, 0x0, 0x40001b8948, 0x4004cc9980, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004cc9950, 0x4004571e00, 0xc03eba3d6f80ae1a, 0x1176f510272, 0x1be4a80, 0x40001b8948, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004cc9950, 0x4004571e00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004e9e5c0, 0x1f, 0x4004cd1d10, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/indexer.getWithAuth(0x4004e9e5c0, 0x1f, 0xe, 0x113846e, 0x11) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:3077 +0x68 github.com/couchbase/indexing/secondary/indexer.getGlobalTopology(0x4004e18320, 0xe, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:2266 +0x78 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).updateNodeList(0x4004e0c400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:220 +0x3c created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:206 +0x98 goroutine 159 [select]: net.(*netFD).connect.func2(0x12f8668, 0x4004d711a0, 0x4004d6c700, 0x4004c81da0, 0x4004c81d40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:118 +0x70 created by net.(*netFD).connect /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:117 +0x1c0 Initializing write barrier = 8000 2021-08-16T13:48:58.968+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:48:58.969+00:00 [Info] Setting ipv6=false 2021-08-16T13:48:58.974+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:48:58.982+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:48:58.982+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:48:58.982+00:00 [Info] Setting log level to Info 2021-08-16T13:48:58.983+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:48:58.986+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:48:58 audit: created new audit service 2021-08-16T13:48:58.995+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:48:58.995+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:48:58.996+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:48:58.996+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:48:58.996+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:48:59.010+00:00 [Info] security context initialized 2021-08-16T13:48:59.010+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:48:59.010+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:48:59.010+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:48:59.010+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:48:59.010+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:48:59.029+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:48:59.032+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:48:59.042+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:48:59.063+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:48:59.065+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:48:59.065+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:48:59.069+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:48:59.069+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:48:59.071+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:48:59.073+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:48:59.078+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:48:59.130+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 51 us 2021-08-16T13:48:59.131+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:48:59.130+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:48:59.132+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:48:59.133+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:48:59.137+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:48:59.138+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:48:59.138+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:48:59.138+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:48:59.138+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:48:59.138+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:59.138+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:59.140+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:48:59.140+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:48:59.140+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:48:59.140+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:59.140+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:48:59.140+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:48:59.140+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:48:59.140+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:48:59.140+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:48:59.140+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:48:59.140+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:48:59.147+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 6.752361ms 2021-08-16T13:48:59.147+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:48:59.148+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:48:59.148+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:48:59.148+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:48:59.149+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:48:59.149+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:48:59.149+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:48:59.162+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:48:59.162+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:48:59.162+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:48:59.162+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:48:59.163+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x40004758df fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x40004758df pc=0xbd8ea4] goroutine 185 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004d609f0 sp=0x4004d609c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004d60a30 sp=0x4004d609f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x4004e70000, 0x4004e78000, 0x11a9890, 0x4004d60b40, 0x4004e70018, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004d60ab0 sp=0x4004d60a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x4004e70000, 0x4004e78000, 0x11a9890, 0x4004d60b40, 0x4004e70018, 0xffff9e0827c0, 0x4004d60c18, 0x49) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004d60af0 sp=0x4004d60ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x40025e2640, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004d60da0 sp=0x4004d60af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x40025e2640, 0x400146b5e0, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004d60e60 sp=0x4004d60da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x40047a1830, 0x40047a1840, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004d61200 sp=0x4004d60e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x40047a1830, 0x40047a1840, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004d618f0 sp=0x4004d61200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004e0bed0, 0x40047f5180, 0x400465cb00, 0x40047a1810) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004d61fb0 sp=0x4004d618f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004d61fb0 sp=0x4004d61fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x4001442000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x4000119a10, 0x4, 0x1136a87, 0xf, 0x4000125e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 34 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x4000126120) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 35 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x4000126138) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 36 [IO wait]: internal/poll.runtime_pollWait(0xffff74dd0098, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000132118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000132100, 0x40000c1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000132100, 0x40000c1000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4001446008, 0x40000c1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x400143c060, 0x40000c6001, 0xdff, 0xdff, 0x400019f8e8, 0x6902d8, 0x40000d3948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x4000126030, 0x40000c6001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x400013e580, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400013e580, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400013e580, 0xf5a960, 0x400143c140, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x400143c120, 0x400012c160, 0x40000d3b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400013a190, 0x12f8f98, 0x400143c120, 0x400012af07, 0x16, 0x40001c6b40, 0x400019fc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400013a190, 0x12f8f98, 0x400143c120, 0x400013a190, 0x400012a160, 0x400012a170, 0x4000132180, 0x400012c000, 0xee0980, 0x40001c6b40, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400013a190, 0x12f8f98, 0x400143c120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4000118990, 0x400019ff78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400243bf78, 0x4000118990, 0x12db8a8, 0x400000e018, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x4000118990, 0x40001423c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x4000118990, 0x40001423c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 37 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 38 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 39 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 40 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 41 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 42 [syscall]: syscall.Syscall(0x3f, 0x0, 0x4000163fa0, 0x4, 0x4000163e68, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x4000163fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x40001ba000, 0x4000163fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x40001b8000, 0x4000163fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 44 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400010c600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 6 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000216600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 49 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400011c800, 0x12c18bc, 0x1, 0x400019de58, 0x4000038060, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x400015dec8, 0x4000038060, 0x400015dec8, 0x679704) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400019dfa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4000080040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 47 [select]: net/http.(*persistConn).readLoop(0x400000c240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 48 [select]: net/http.(*persistConn).writeLoop(0x400000c240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 66 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4000080040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 88 [select]: net/http.(*persistConn).writeLoop(0x400000cea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 70 [IO wait]: internal/poll.runtime_pollWait(0xffff74dcfec8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40023ce298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40023ce280, 0x40023ef000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40023ce280, 0x40023ef000, 0x1000, 0x1000, 0x400006bbe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001280a8, 0x40023ef000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c6c0, 0x40023ef000, 0x1000, 0x1000, 0x400006bcf8, 0x67867c, 0x4000154b40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400041dc20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x400041dc20, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000c6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 69 [IO wait]: internal/poll.runtime_pollWait(0xffff74dcffb0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40023ce118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40023ce100, 0x40023d8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40023ce100, 0x40023d8000, 0x1000, 0x1000, 0x4000069a88, 0x68f7a4, 0x40023cc890) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000128048, 0x40023d8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c240, 0x40023d8000, 0x1000, 0x1000, 0xf12180, 0x4000180220, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400041d440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400041d440, 0xa, 0x1008880, 0x40001801e0, 0x199, 0xad9, 0x4000069b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400041d440, 0x2, 0x0, 0x9, 0x56b, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400044d140) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400044d140, 0x40023ee000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40001d9700, 0x40023ee000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40001d9700, 0x40023ee000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001d9740, 0x40023ee000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40023cc840, 0x0, 0x4000069eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40023cc840, 0x0, 0x0, 0x4000069f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40023cc840, 0xee0d40, 0x40001801e0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40000bc630, 0x400041db60, 0x4000154a20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 71 [select]: net/http.(*persistConn).writeLoop(0x400000c6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 22 [select]: net/http.(*persistConn).readLoop(0x400000d560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 23 [select]: net/http.(*persistConn).writeLoop(0x400000d560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 28 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x4002486380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 30 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x4002486380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 29 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x4002486380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 8 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x4002486380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 167 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4004e020b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 166 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004e020b0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 165 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004e020b0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 11 [select]: net/http.(*persistConn).readLoop(0x40001c7b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 12 [select]: net/http.(*persistConn).writeLoop(0x40001c7b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 33 [IO wait]: internal/poll.runtime_pollWait(0xffff74dcf788, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000132e18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000132e00, 0x4002596000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000132e00, 0x4002596000, 0x1000, 0x1000, 0xffff6d611700, 0x100100000000, 0xffff6d7eafff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4001446270, 0x4002596000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c7b00, 0x4002596000, 0x1000, 0x1000, 0x40044ae180, 0x200000003, 0x40044ae180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400143cd20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400143cd20, 0x40023f600a, 0x400019b8c8, 0x4d3b10, 0x40001ba060, 0x40023f6000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400143cd20, 0x0, 0x1, 0x0, 0x40001ba078, 0x40023f6000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40025609c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40025609c0, 0x40044c8000, 0x1000, 0x1000, 0x400019b9b8, 0x0, 0x400019b988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000216380, 0x40044c8000, 0x1000, 0x1000, 0x4001428180, 0x3, 0x400019bbe0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000216380, 0x40044c8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40002163c0, 0x40044c8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400019bdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400019bdd0, 0xa, 0x40044c8201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400019bdd0, 0x40044c820a, 0x1, 0x4002521dc9, 0x0, 0x40044c82e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400019bdd0, 0x400450a40a, 0x4002521dc9, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40001393c0, 0x40001c49c0, 0x17, 0x400019bf20, 0x400000efa8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x40001393c0, 0x113075a, 0x7, 0x400000efa8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x4000139380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 82 [IO wait]: internal/poll.runtime_pollWait(0xffff74dcf958, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40023ced98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40023ced80, 0x4000214000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40023ced80, 0x4000214000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b80a8, 0x4000214000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000d560, 0x4000214000, 0x1000, 0x1000, 0x40044ae300, 0x200000003, 0x40044ae300) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40000821e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000821e0, 0x40023f600a, 0x400019f8c8, 0x4d3b10, 0x40001ba060, 0x40023f6000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40000821e0, 0x0, 0x1, 0x0, 0x40001ba078, 0x40023f6000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400145e870) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400145e870, 0x40025ba000, 0x1000, 0x1000, 0x400019f9b8, 0x0, 0x400019f988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4001468400, 0x40025ba000, 0x1000, 0x1000, 0x4001428180, 0x3, 0x400019fbe0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4001468400, 0x40025ba000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4001468440, 0x40025ba000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400019fdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400019fdd0, 0xa, 0x40025ba201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400019fdd0, 0x40025ba20a, 0x1, 0x4000036a4b, 0x0, 0x40025ba263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400019fdd0, 0x400146c20a, 0x4000036a4b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40001393c0, 0x40001d4240, 0x24, 0x400019ff20, 0x400000ef90, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x40001393c0, 0x113075a, 0x7, 0x400000ef90, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x4000139380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 164 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4001469980) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 114 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000080980) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 55 [IO wait]: internal/poll.runtime_pollWait(0xffff74dcf5b8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400024e098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400024e080, 0x4002448000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400024e080, 0x4002448000, 0x1000, 0x1000, 0x40023d3688, 0x69f178, 0x40023d36f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4001446038, 0x4002448000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c6360, 0x4002448000, 0x1000, 0x1000, 0x5, 0x40000f2500, 0x40023d37c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400041df80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400041df80, 0x68f90a, 0x4000222420, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400041df80, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4002560c90) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4002560c90, 0x40044ce000, 0x1000, 0x1000, 0x40023d3968, 0x44d8a8, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40002164c0, 0x40044ce000, 0x1000, 0x1000, 0x40001ba4d8, 0x1, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40002164c0, 0x40044ce000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000216500, 0x40044ce000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40023d3d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40023d3d80, 0xa, 0x40044ce201, 0x1, 0xda7, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40023d3d80, 0x40044ce20a, 0x1, 0x40001b57b8, 0x0, 0x40044ce259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40023d3d80, 0x400450530a, 0x40001b57b8, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40001393c0, 0x4002446000, 0x1f, 0x40023d3ee0, 0x400254e000, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x40001393c0, 0x113075a, 0x7, 0x4002521de0, 0xd, 0x400254e000, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x4000139380, 0x4002521de0, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 80 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000130f40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 87 [select]: net/http.(*persistConn).readLoop(0x400000cea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 91 [IO wait]: internal/poll.runtime_pollWait(0xffff74dcf870, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004772498, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4004772480, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4004772480, 0x12d9d48, 0x4000080a00, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c4ee40, 0x0, 0x4004506600, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c4ee40, 0x4002521b00, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x40001d8580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 76 [select]: net/http.(*persistConn).readLoop(0x40001c6360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 77 [select]: net/http.(*persistConn).writeLoop(0x40001c6360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 81 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x400244a300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 98 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x4002598570) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 99 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x400024e100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 100 [IO wait]: internal/poll.runtime_pollWait(0xffff74dcf4d0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400024e218, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400024e200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400024e200, 0x0, 0x400142d698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x400254e2b8, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x400254e2b8, 0x12ed9d0, 0x400254e2b8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x400024e180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 101 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400144a750, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 102 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400144a750, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 103 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x400144a750) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 104 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x4002599050) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 163 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40025e0540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 108 [IO wait]: internal/poll.runtime_pollWait(0xffff74dcf3e8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400024e398, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400024e380, 0x4002451000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400024e380, 0x4002451000, 0x1000, 0x1000, 0x4002442be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4001446160, 0x4002451000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c6480, 0x4002451000, 0x1000, 0x1000, 0x4002442cf8, 0x67867c, 0x400143f200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400244ac60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x400244ac60, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40001c6480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 107 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40002ec700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 109 [select]: net/http.(*persistConn).writeLoop(0x40001c6480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 119 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x4000143d60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 90 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000081680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 89 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001314c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 92 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x4000131ec0, 0x40023db260, 0x40023db2d0, 0x4004690d00, 0x12e8f90, 0x4000200c40, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x4004c507c4, 0x5, 0x40001d8580, 0x12eca70, 0x4000081f00, 0x130d478, 0x40001f80e0, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40002010a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40002010a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 120 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x4000143d60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 121 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x4000131ec0, 0x4004690d00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 122 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x4000200c40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 123 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x4000200c40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 124 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x400243d290, 0x40046da540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 125 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x4000080940) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 126 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x4000143e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 127 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x4000143e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 128 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x40014464e8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 129 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x4000143e00, 0x40046da5a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 130 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400011c800, 0x1144559, 0x1b, 0x40023d1e38, 0x40046da6c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004691820, 0x40046da6c0, 0xf1f700, 0x400047c080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40023d1f80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004693200, 0x4004691820) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 131 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 132 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400013b400, 0xd00007, 0x4004cf6c38, 0x5, 0x4004691830, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 133 [IO wait]: internal/poll.runtime_pollWait(0xffff74dcf300, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400024f518, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400024f500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400024f500, 0xffff9e0765b8, 0x30, 0x4004d14b40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40046f5ae8, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40046f5ae8, 0x4004d14b40, 0x11ae500, 0x4002486940, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40024868c0, 0x12ed9d0, 0x40046f5ae8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40024868c0, 0x12ed9d0, 0x40046f5ae8, 0x4001442000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 134 [IO wait]: internal/poll.runtime_pollWait(0xffff74dcf218, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400024f618, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400024f600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400024f600, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40046f5b00, 0xffff9e070030, 0xffff741a3d50, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40046f5b00, 0x665e1c, 0x0, 0xfe5e60, 0x4004d20dc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x40046f5b18, 0x4004d14bd0, 0x11ae500, 0x4002486a20, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40024869a0, 0x12ec710, 0x40046f5b18, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40024869a0, 0x12ec710, 0x40046f5b18, 0x4001442000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 162 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40025e0540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 15 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40025e0540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 60 [IO wait]: internal/poll.runtime_pollWait(0xffff74098380, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40023ce918, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40023ce900, 0x40024b5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40023ce900, 0x40024b5000, 0x1000, 0x1000, 0x40000b6be0, 0x469b10, 0x4004685e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b8610, 0x40024b5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40046d0fc0, 0x40024b5000, 0x1000, 0x1000, 0x40000b6cf8, 0x67867c, 0x4004729bc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400143d920) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x400143d920, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40046d0fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 61 [select]: net/http.(*persistConn).writeLoop(0x40046d0fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 58 [IO wait]: internal/poll.runtime_pollWait(0xffff74dcf6a0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000133218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000133200, 0x4004519000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000133200, 0x4004519000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000128218, 0x4004519000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000cea0, 0x4004519000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001bb320) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001bb320, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40001bb320, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40025617d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40025617d0, 0x40045bc200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000217180, 0x40045bc200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000217180, 0x40045bc200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40002171c0, 0x40045bc200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40044ed760, 0x0, 0x40000546b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40044ed760, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40044ed760, 0xee0d40, 0x400008c640, 0x12db068, 0x40002171c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40001575f0, 0x400143d8c0, 0x40045075c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 14 [semacquire]: sync.runtime_Semacquire(0x4004e0bed8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004e0bed0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x400465cb00, 0x400044cd50, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffcdfdc969, 0x29, 0xffffcdfdc9cb, 0x25, 0x4004e188a0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004c514b0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffcdfdc969, 0x29, 0xffffcdfdc9cb, 0x25, 0x4004e188a0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004c514b0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004d61190, 0x4004d60e60, 0x400044cd50, 0x400022d200, 0x4000216600, 0x0, 0x1bf, 0x4002436da8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x4001442000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004c514b0, 0xb, 0x4004c514a8, 0x6, 0x4004c514c0, 0xd, 0x4004c4d420, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x4001442000, 0x4000410000, 0x4001440030) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x4001442000, 0x400047c600, 0x2, 0x2, 0x400047c610, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 16 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40025e0540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 149 [IO wait]: internal/poll.runtime_pollWait(0xffff74098468, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400147eb18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400147eb00, 0x40047f3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400147eb00, 0x40047f3000, 0x1000, 0x1000, 0x40045c0ad8, 0x412bc0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000102a0, 0x40047f3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400147cd80, 0x40047f3000, 0x1000, 0x1000, 0x40045d4120, 0x60, 0x58) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400479b020) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400479b020, 0xa, 0x0, 0x0, 0x0, 0x0, 0x4000058c38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400479b020, 0x0, 0x4000058c78, 0x655784, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40045e8120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40045e8120, 0x40045bd000, 0x200, 0x200, 0x4, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000217640, 0x40045bd000, 0x200, 0x200, 0x4000035807, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000217640, 0x40045bd000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000217680, 0x40045bd000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40045d31e0, 0x0, 0x4000058eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40045d31e0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40045d31e0, 0xee0d40, 0x400008c8c0, 0x12db068, 0x4000217680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4000157cb0, 0x40045d4240, 0x40045d8300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 168 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40025e0540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 169 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400011c800, 0x1144559, 0x1b, 0x4002439e38, 0x400479d500, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x40047a0e90, 0x400479d500, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4002439f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x400479aa20, 0x40047a0e90) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 174 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40014699c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 172 [select]: net/http.(*persistConn).readLoop(0x400147cd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 173 [select]: net/http.(*persistConn).writeLoop(0x400147cd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 175 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x400145def0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 176 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x400145def0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 177 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x400145def0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 178 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x400145def0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 179 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x40047ef300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 180 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x40047ef300, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 181 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x4001442000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 182 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x4001442000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 183 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x4001442000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 184 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:49:04.388+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:49:04.388+00:00 [Info] Setting ipv6=false 2021-08-16T13:49:04.403+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:49:04.410+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:49:04.410+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:49:04.410+00:00 [Info] Setting log level to Info 2021-08-16T13:49:04.410+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:49:04.414+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:49:04 audit: created new audit service 2021-08-16T13:49:04.422+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:49:04.422+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:49:04.422+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:49:04.422+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:49:04.422+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:49:04.434+00:00 [Info] security context initialized 2021-08-16T13:49:04.434+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:49:04.434+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:49:04.434+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:49:04.434+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:49:04.434+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:49:04.447+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:49:04.448+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:49:04.455+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:49:04.472+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:49:04.472+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:49:04.473+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:49:04.473+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:49:04.475+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:49:04.476+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:49:04.477+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:49:04.477+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:49:04.527+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:49:04.527+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 151 us 2021-08-16T13:49:04.527+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:49:04.530+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:49:04.530+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:49:04.534+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:49:04.534+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:49:04.534+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:49:04.534+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:49:04.534+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:49:04.534+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:04.534+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:04.536+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:49:04.536+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:49:04.536+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:49:04.536+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:04.537+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:49:04.537+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:49:04.537+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:49:04.537+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:49:04.537+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:49:04.537+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:49:04.537+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:49:04.542+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 4.659879ms 2021-08-16T13:49:04.542+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:49:04.543+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:49:04.543+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:04.543+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:49:04.543+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:49:04.543+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:49:04.545+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:49:04.553+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:49:04.554+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:49:04.554+00:00 [Info] RebalanceMgr::initService Init 2021-08-16T13:49:04.554+00:00 [Info] ServiceMgr::registerWithServer nodeuuid 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:04.556+00:00 [Info] ServiceMgr::GetTaskList [] 2021-08-16T13:49:04.556+00:00 [Info] ServiceMgr::GetTaskList returns &{[0 0 0 0 0 0 0 0] []} 2021-08-16T13:49:04.557+00:00 [Info] ServiceMgr::GetCurrentTopology [] 2021-08-16T13:49:04.557+00:00 [Info] ServiceMgr::GetCurrentTopology returns &{[0 0 0 0 0 0 0 0] [8a95cde683482bdec453edbb8b18eb5d] true []} 2021-08-16T13:49:04.557+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:49:04.559+00:00 [Info] ServiceMgr::GetCurrentTopology [0 0 0 0 0 0 0 0] 2021-08-16T13:49:04.559+00:00 [Info] ServiceMgr::GetTaskList [0 0 0 0 0 0 0 0] 2021-08-16T13:49:04.564+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:49:04.565+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x400023f21f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x400023f21f pc=0xbd8ea4] goroutine 159 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004d089f0 sp=0x4004d089c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004d08a30 sp=0x4004d089f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x40000a9a00, 0x4004d50600, 0x11a9890, 0x4004d08b40, 0x40000a9a18, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004d08ab0 sp=0x4004d08a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x40000a9a00, 0x4004d50600, 0x11a9890, 0x4004d08b40, 0x40000a9a18, 0xffff956086e0, 0x4004d08c18, 0x49) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004d08af0 sp=0x4004d08ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4004d48140, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004d08da0 sp=0x4004d08af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4004d48140, 0x400007efc0, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004d08e60 sp=0x4004d08da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004e9a3c0, 0x4004e9a3d0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004d09200 sp=0x4004d08e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004e9a3c0, 0x4004e9a3d0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004d098f0 sp=0x4004d09200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4002561f30, 0x4004ec8000, 0x4004d44580, 0x4004e9a3a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004d09fb0 sp=0x4004d098f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004d09fb0 sp=0x4004d09fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x40001fa000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x400008d770, 0x4, 0x1136a87, 0xf, 0x4000095e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 5 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400000e108) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 6 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400000e120) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 7 [IO wait]: internal/poll.runtime_pollWait(0xffff6c476018, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000b2718, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000b2700, 0x40000e5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000b2700, 0x40000e5000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000100b0, 0x40000e5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x4000082840, 0x400021e001, 0xdff, 0xdff, 0x400019d8e8, 0x6902d8, 0x4000091948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400000e168, 0x400021e001, 0xdff, 0xdff, 0x40000919f8, 0x6f1968, 0x4000091938) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40001e6580, 0xa, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40001e6580, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40001e6580, 0xf5a960, 0x40001ba380, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x40001ba360, 0x40001c22a0, 0x4000091b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x4000180190, 0x12f8f98, 0x40001ba360, 0x40001b4c37, 0x16, 0x40001c7320, 0x400019dc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x4000180190, 0x12f8f98, 0x40001ba360, 0x4000180190, 0x40001b4448, 0x40001b4450, 0x4000208080, 0x40001c21c0, 0xee0980, 0x40001c7320, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x4000180190, 0x12f8f98, 0x40001ba360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x400008c570, 0x400019df78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400019ff78, 0x400008c570, 0x12db8a8, 0x400000e138, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x400008c570, 0x40000a0460, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x400008c570, 0x40000a0460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 8 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 9 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 10 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 11 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 12 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 34 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400010c200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 13 [syscall]: syscall.Syscall(0x3f, 0x0, 0x4000438fa0, 0x4, 0x4000438e68, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x4000438fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x40001ba000, 0x4000438fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x40001b8000, 0x4000438fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 23 [select]: net/http.(*persistConn).writeLoop(0x40001c6480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 31 [select]: net/http.(*persistConn).writeLoop(0x40001c65a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 22 [select]: net/http.(*persistConn).readLoop(0x40001c6480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 52 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400006c800, 0x12c18bc, 0x1, 0x4000475e58, 0x4000182780, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x400043aec8, 0x4000182780, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000475fa0, 0x4000122080, 0xe) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4000124000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 53 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4000124000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 40 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000080f80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 38 [IO wait]: internal/poll.runtime_pollWait(0xffff6c475e48, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000126318, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000126300, 0x4000164000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000126300, 0x4000164000, 0x1000, 0x1000, 0x400142ebe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000162000, 0x4000164000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c6900, 0x4000164000, 0x1000, 0x1000, 0x400142ecf8, 0x67867c, 0x4000230420) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40000822a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40000822a0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40001c6900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 39 [select]: net/http.(*persistConn).writeLoop(0x40001c6900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 14 [IO wait]: internal/poll.runtime_pollWait(0xffff6c475f30, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000126198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000126180, 0x40000e4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000126180, 0x40000e4000, 0x1000, 0x1000, 0x4000065a88, 0x68f7a4, 0x40001e6b50) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010008, 0x40000e4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c6480, 0x40000e4000, 0x1000, 0x1000, 0xf12180, 0x4000180270, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003eccc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003eccc0, 0xa, 0x1008880, 0x4000180230, 0x199, 0x1000, 0x4000065b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003eccc0, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40001190b0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40001190b0, 0x4000014000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000080140, 0x4000014000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000080140, 0x4000014000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000080180, 0x4000014000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40001e6b00, 0x0, 0x4000065eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40001e6b00, 0x0, 0x0, 0x4000065f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40001e6b00, 0xee0d40, 0x4000180230, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40000af680, 0x40001ba780, 0x4000038600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 63 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4000179b80, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 62 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4000179b80, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 142 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x4004511500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 15 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x40002009a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 72 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000081480) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 30 [select]: net/http.(*persistConn).readLoop(0x40001c65a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 67 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x40002009a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 68 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x40002009a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 69 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x40002009a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 120 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x40023e5180, 0x400467ed80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 64 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4000179b80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 58 [select]: net/http.(*persistConn).readLoop(0x40001c6d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 59 [select]: net/http.(*persistConn).writeLoop(0x40001c6d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 73 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x40003ecf60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 74 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x4000228300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 75 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x4000208000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 76 [IO wait]: internal/poll.runtime_pollWait(0xffff6c4758d8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000208218, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000208200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000208200, 0x0, 0x4000054698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x400012c348, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x400012c348, 0x12ed9d0, 0x400012c348, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x4000208100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 77 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400242e8f0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 78 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400242e8f0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 79 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x400242e8f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 80 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x40002294a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 86 [IO wait]: internal/poll.runtime_pollWait(0xffff6c475538, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000208518, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000208500, 0x4002430000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000208500, 0x4002430000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011c138, 0x4002430000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400041e240, 0x4002430000, 0x1000, 0x1000, 0x4000433e00, 0x200000003, 0x4000433e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40024fc000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40024fc000, 0x400001c00a, 0x40004718c8, 0x4d3b10, 0x40001ba060, 0x400001c000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40024fc000, 0x0, 0x1, 0x0, 0x40001ba078, 0x400001c000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40024facf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40024facf0, 0x400442f000, 0x1000, 0x1000, 0x40004719b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000124f80, 0x400442f000, 0x1000, 0x1000, 0x40001165f8, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000124f80, 0x400442f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000124fc0, 0x400442f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000471dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000471dd0, 0xa, 0x400442f201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000471dd0, 0x400442f20a, 0x1, 0x40045e6ae9, 0x0, 0x400442f2e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000471dd0, 0x40045a4a0a, 0x40045e6ae9, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400242ea00, 0x4000128720, 0x17, 0x4000471f20, 0x400012c570, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x400242ea00, 0x113075a, 0x7, 0x400012c570, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x400242e9c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 119 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x400471c140) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 83 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40000fc5b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 96 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001241c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 87 [IO wait]: internal/poll.runtime_pollWait(0xffff6c475708, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000126418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000126400, 0x400143e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000126400, 0x400143e000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b8400, 0x400143e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c6d80, 0x400143e000, 0x1000, 0x1000, 0x4001414900, 0x200000003, 0x4001414900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000117080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000117080, 0x400001c00a, 0x400019b8c8, 0x4d3b10, 0x40001ba060, 0x400001c000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000117080, 0x0, 0x1, 0x0, 0x40001ba078, 0x400001c000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40024faa80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40024faa80, 0x400442e000, 0x1000, 0x1000, 0x400019b9b8, 0x0, 0x400019b988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000124e80, 0x400442e000, 0x1000, 0x1000, 0x4000187200, 0x3, 0x400019bbe0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000124e80, 0x400442e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000124ec0, 0x400442e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400019bdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400019bdd0, 0xa, 0x400442e201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400019bdd0, 0x400442e20a, 0x1, 0x40045e689b, 0x0, 0x400442e263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400019bdd0, 0x40001c290a, 0x40045e689b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400242ea00, 0x400011e390, 0x24, 0x400019bf20, 0x400012c558, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x400242ea00, 0x113075a, 0x7, 0x400012c558, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x400242e9c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 117 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40023e4600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 90 [select]: net/http.(*persistConn).readLoop(0x400041e240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 91 [select]: net/http.(*persistConn).writeLoop(0x400041e240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 107 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40023e4d00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 118 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x400471c140) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 116 [select]: net/http.(*persistConn).writeLoop(0x400041f200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 98 [IO wait]: internal/poll.runtime_pollWait(0xffff6c475280, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000127198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000127180, 0x4004431000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000127180, 0x4004431000, 0x1000, 0x1000, 0x4004421688, 0x69f178, 0x40044216f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b8698, 0x4004431000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c7c20, 0x4004431000, 0x1000, 0x1000, 0x5, 0x40044da000, 0x40044217c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001bae40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001bae40, 0x68f90a, 0x40001dcbb0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40001bae40, 0x0, 0x4004421868, 0x41a0a8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40024fb0e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40024fb0e0, 0x4004439000, 0x1000, 0x1000, 0x4004421998, 0x44e884, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000125140, 0x4004439000, 0x1000, 0x1000, 0x40001165f8, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000125140, 0x4004439000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000125180, 0x4004439000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4004421d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004421d80, 0xa, 0x4004439201, 0x1, 0xffff6c4b7020, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4004421d80, 0x400443920a, 0x1, 0x400244d600, 0x0, 0x4004439259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4004421d80, 0x40045e0d0a, 0x400244d600, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400242ea00, 0x40045d90c0, 0x1f, 0x4004421ee0, 0x40045dec78, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x400242ea00, 0x113075a, 0x7, 0x40045e6b00, 0xd, 0x40045dec78, 0x0, 0x0, 0x2) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x400242e9c0, 0x40045e6b00, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 115 [select]: net/http.(*persistConn).readLoop(0x400041f200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 101 [select]: net/http.(*persistConn).readLoop(0x40001c7c20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 102 [select]: net/http.(*persistConn).writeLoop(0x40001c7c20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 108 [IO wait]: internal/poll.runtime_pollWait(0xffff6c475450, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000127698, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000127680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000127680, 0x12d9d48, 0x40001250c0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c387f8, 0x0, 0x40002307e0, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c387f8, 0x40001b4b00, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x4000125ec0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 109 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x40023e5180, 0x4001435180, 0x40014351f0, 0x400467ed80, 0x12e8f90, 0x40045110a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x4004c422e4, 0x5, 0x4000125ec0, 0x12eca70, 0x4000125d80, 0x130d478, 0x400009e070, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x4004466540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x4004466540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 121 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40045110a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 122 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40045110a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 123 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x400141e090, 0x40047280c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 124 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x4000124180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 125 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x400471c1e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 126 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x400471c1e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 127 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x400011c5b8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 128 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x400471c1e0, 0x4004728120) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 129 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400006c800, 0x1144559, 0x1b, 0x400250be38, 0x4004728240, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x400467f8a0, 0x4004728240, 0xf1f700, 0x400046c080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400250bf80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x400471ac00, 0x400467f8a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 130 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 131 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400012ef50, 0xd00007, 0x4004758658, 0x5, 0x400467f8b0, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 132 [IO wait]: internal/poll.runtime_pollWait(0xffff6c475368, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000209b18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000209b00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000209b00, 0x0, 0x400142bd88, 0x665fe4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x400475a2b8, 0x611a6cd0, 0x400142bdb8, 0x445c24) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x400475a2b8, 0x400142be20, 0x400142be28, 0x18, 0x4000001b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x4004511180, 0x12ed9d0, 0x400475a2b8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x4004511180, 0x12ed9d0, 0x400475a2b8, 0x40001fa000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 133 [IO wait]: internal/poll.runtime_pollWait(0xffff6c475198, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000209c18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000209c00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000209c00, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x400475a2d0, 0xffff95787108, 0xffff6c4cf5f0, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x400475a2d0, 0x665e1c, 0x0, 0xfe5e60, 0x4002535dc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x400475a2e8, 0x4004752de0, 0x11ae500, 0x40045112e0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x4004511260, 0x12ec710, 0x400475a2e8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x4004511260, 0x12ec710, 0x400475a2e8, 0x40001fa000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 111 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).processSchedIndexes(0x4004718dc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:296 +0x44 created by github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:218 +0x3c goroutine 141 [semacquire]: sync.runtime_Semacquire(0x4002561f38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4002561f30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x4004d44580, 0x400041c750, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xfffff4d14969, 0x29, 0xfffff4d149cb, 0x25, 0x4002565080, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004c42fd0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xfffff4d14969, 0x29, 0xfffff4d149cb, 0x25, 0x4002565080, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004c42fd0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004d09190, 0x4004d08e60, 0x400041c750, 0x40023d8240, 0x4000080f80, 0x0, 0x1bf, 0x400477ada8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x40001fa000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004c42fd0, 0xb, 0x4004c42fc8, 0x6, 0x4004c42fe0, 0xd, 0x4004c37c80, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x40001fa000, 0x40003e0000, 0x40001182a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x40001fa000, 0x4004410390, 0x2, 0x2, 0x40044103a0, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 45 [IO wait]: internal/poll.runtime_pollWait(0xffff6c1e2958, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000b3218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000b3200, 0x4004d0a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000b3200, 0x4004d0a000, 0x1000, 0x1000, 0x400476cbe0, 0x469b10, 0x400476a480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000102a8, 0x4004d0a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004716fc0, 0x4004d0a000, 0x1000, 0x1000, 0x400476ccf8, 0x67867c, 0x4002564600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002479260) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4002479260, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004716fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 46 [select]: net/http.(*persistConn).writeLoop(0x4004716fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 147 [IO wait]: internal/poll.runtime_pollWait(0xffff6c1e2a40, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000407198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000407180, 0x4004dd0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000407180, 0x4004dd0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001622c0, 0x4004dd0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003d9560, 0x4004dd0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004dc64e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004dc64e0, 0xa, 0x4000170123, 0x11320c7, 0x9, 0x1, 0x4002539438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004dc64e0, 0xd4fa28, 0x4002539478, 0x655784, 0x11320c7, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400254f0b0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400254f0b0, 0x4002551e00, 0x200, 0x200, 0x4, 0x0, 0x400011e810) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000081a80, 0x4002551e00, 0x200, 0x200, 0x40003d7f47, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000081a80, 0x4002551e00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000081ac0, 0x4002551e00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4002563340, 0x0, 0x40025396b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4002563340, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4002563340, 0xee0d40, 0x4000180be0, 0x12db068, 0x4000081ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4002576510, 0x40024794a0, 0x4002564a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 110 [IO wait]: internal/poll.runtime_pollWait(0xffff6c475620, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000209098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000209080, 0x40024f3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000209080, 0x40024f3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011c320, 0x40024f3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400041f200, 0x40024f3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40024fd560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40024fd560, 0xa, 0x0, 0x0, 0x0, 0x0, 0x400043a438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40024fd560, 0x0, 0x400043a478, 0x655784, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004c3ef00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004c3ef00, 0x4004c01000, 0x200, 0x200, 0x4, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000125fc0, 0x4004c01000, 0x200, 0x200, 0x40001cf887, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000125fc0, 0x4004c01000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004c6e000, 0x4004c01000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004c3ac60, 0x0, 0x400043a6b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004c3ac60, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004c3ac60, 0xee0d40, 0x4000098e60, 0x12db068, 0x4004c6e000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004c3c6c0, 0x400240ccc0, 0x4002470c60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 143 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x4004511500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 144 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x4004511500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 145 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x4004511500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 162 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x40023e5c80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 65 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x4004511500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 178 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400006c800, 0x1144559, 0x1b, 0x4002505e38, 0x4004729440, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004bb4ab0, 0x4004729440, 0x400041e240, 0x4000054640) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x40024a6ae0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4002505f80, 0x12c2e00, 0x1000000010000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x400471bce0, 0x4004bb4ab0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 184 [select]: net/http.(*persistConn).writeLoop(0x4004717440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 181 [select]: net/http.(*persistConn).readLoop(0x40003d9560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 182 [select]: net/http.(*persistConn).writeLoop(0x40003d9560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 148 [runnable]: net/http.(*persistConn).roundTrip(0x40001c65a0, 0x4004e52140, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004e3a500, 0xf8, 0x10a8660, 0x4004e60e78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004e3a500, 0x1b6e900, 0xc03eb9322134bd33, 0x1bfae73b2b) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004e3a400, 0x12db048, 0x1b6e900, 0xc03eb9322134bd33, 0x1bfae73b2b, 0x1be4a80, 0x0, 0x400011c828, 0x4004e233e0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004e233b0, 0x4004e3a400, 0xc03eb9322134bd33, 0x1bfae73b2b, 0x1be4a80, 0x400011c828, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004e233b0, 0x4004e3a400, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004e3c7e0, 0x5c, 0x4004e61640, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004c3d0e0, 0x4000253770, 0x47, 0x12db908, 0x40001c3500, 0xeddd00, 0x400475b140, 0x4004e61640, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.(*Pool).Refresh(0x4004c48b60, 0x40000358c0, 0x34) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:689 +0x3c4 github.com/couchbase/indexing/secondary/dcp.(*Client).GetPool(0x4004764bb0, 0x113075a, 0x7, 0x12db908, 0x40001c3500, 0x4004c3f950, 0x40001c5bf0, 0x15, 0x1, 0x4004f30100, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:831 +0x204 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x20, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:296 +0x440 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004e61d40, 0x4, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x4004764b40, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x4004764b40, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40023e5cc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1267 +0xf8 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 149 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x4004718dc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:224 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 150 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x4004718dc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:612 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 151 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x4004718dc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:662 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 152 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x4004718dc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:752 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 153 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4002578700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:249 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 185 [select]: net/http.(*persistConn).roundTrip(0x40047177a0, 0x4004e52480, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004e3ab00, 0xf8, 0x10a8660, 0x4004e4c558) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004e3ab00, 0x1b6e900, 0xc03eb93221a774b4, 0x1bfb59f2ad) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004e3aa00, 0x12db048, 0x1b6e900, 0xc03eb93221a774b4, 0x1bfb59f2ad, 0x1be4a80, 0x0, 0x400011c870, 0x4004e23a70, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004e23a40, 0x4004e3aa00, 0xc03eb93221a774b4, 0x1bfb59f2ad, 0x1be4a80, 0x400011c870, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004e23a40, 0x4004e3aa00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004e441c0, 0x33, 0x4004e4ccd0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004e28b40, 0x4004fbcb80, 0x1e, 0x12db908, 0x40001217e0, 0x10f78c0, 0x4004e62f20, 0x4004e4ccd0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.(*Pool).getTerseBucket(0x4004b9aa90, 0x4004e7c110, 0xd, 0x12db908, 0x40001217e0, 0xeddd00, 0x400475b5c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:620 +0x104 github.com/couchbase/indexing/secondary/dcp.(*Pool).Refresh(0x4004b9aa90, 0x4004d96380, 0x34) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:694 +0xe4 github.com/couchbase/indexing/secondary/dcp.(*Client).GetPool(0x4004765270, 0x113075a, 0x7, 0x12db908, 0x40001217e0, 0x40046c1bf0, 0x400017f2a8, 0x15, 0x1, 0x4004dbd1c0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:831 +0x204 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x4004e22c90, 0xffff6c4cf2c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:296 +0x440 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004e4d4b0, 0x1063520, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x4004765200, 0x4004765200, 0x40001b2880) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.FetchNewClusterInfoCache(0xfffff4d1494e, 0xe, 0x113075a, 0x7, 0x1130e44, 0x8, 0x4004de00c4, 0x2, 0xf01) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:123 +0x134 github.com/couchbase/indexing/secondary/manager.(*IndexManager).FetchNewClusterInfoCache(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:306 github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).getIndexMetadata(0x1be5520, 0x12ec7a0, 0x40023e5f80, 0x400012f950, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:1284 +0x74 github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).handleIndexMetadataRequest(0x1be5520, 0x12edbe0, 0x40045115e0, 0x4004bbec00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:1271 +0x3e8 net/http.HandlerFunc.ServeHTTP(0x400467f7a0, 0x12edbe0, 0x40045115e0, 0x4004bbec00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2069 +0x40 net/http.(*ServeMux).ServeHTTP(0x40023e5480, 0x12edbe0, 0x40045115e0, 0x4004bbec00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2448 +0x190 net/http.serverHandler.ServeHTTP(0x4004511180, 0x12edbe0, 0x40045115e0, 0x4004bbec00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2887 +0xbc net/http.(*conn).serve(0x4000130d20, 0x12f86a0, 0x4000167780) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:1952 +0x71c created by net/http.(*Server).Serve /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:3013 +0x318 goroutine 155 [select]: net/http.(*persistConn).roundTrip(0x4004717680, 0x4000167a00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004bbf000, 0xf8, 0x10a8660, 0x4004e5ce38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004bbf000, 0x1b6e900, 0xc03eb932216c4b14, 0x1bfb1ec915) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004bbef00, 0x12db048, 0x1b6e900, 0xc03eb932216c4b14, 0x1bfb1ec915, 0x1be4a80, 0x0, 0x4000162300, 0x40046c1ef0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x40046c1ec0, 0x4004bbef00, 0xc03eb932216c4b14, 0x1bfb1ec915, 0x1be4a80, 0x4000162300, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x40046c1ec0, 0x4004bbef00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004dde4e0, 0x5c, 0x4004e5d600, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004e286c0, 0x4000404ff0, 0x47, 0x12db908, 0x40001217a0, 0xeddd00, 0x4004df2270, 0x4004e5d600, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.(*Pool).Refresh(0x400470be10, 0x4004707d00, 0x34) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:689 +0x3c4 github.com/couchbase/indexing/secondary/dcp.(*Client).GetPool(0x4004765030, 0x113075a, 0x7, 0x12db908, 0x40001217a0, 0x4004e23500, 0x4004e54168, 0x15, 0x1, 0x4004e02880, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:831 +0x204 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x240, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:296 +0x440 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004e5dd00, 0x4004e22960, 0xffff6c4cf2c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x4004764fc0, 0xd47600, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x4004764fc0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x40001fa000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9702 +0x344 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 163 [IO wait]: internal/poll.runtime_pollWait(0xffff6c1e2788, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004e04518, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004e04500, 0x4004e38000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004e04500, 0x4004e38000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011c708, 0x4004e38000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x4004e36180, 0x4004c77801, 0x5ff, 0x5ff, 0x4004dcb888, 0x6902d8, 0x4004dcb8e8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400475acd8, 0x4004c77801, 0x5ff, 0x5ff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x4004e1fa20, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004e1fa20, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004e1fa20, 0xf5a960, 0x4004e36f80, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4004e36f60, 0x40001c3760, 0x4004dcbac8, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400012fa40, 0x12f8f98, 0x4004e36f60, 0x4000099400, 0x16, 0x199, 0x4004dcbc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400012fa40, 0x12f8f98, 0x4004e36f60, 0x400012fa40, 0x4004759960, 0x4004759970, 0x4004e04900, 0x40001c3620, 0xfb91e0, 0x40001c3740, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400012fa40, 0x12f8f98, 0x4004e36f60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4004e227e0, 0x4004dcbf10, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x4004dcbf10, 0x4004e227e0, 0x0, 0x0, 0x0, 0xd8d03c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth/service.RegisterManager(0x13007d8, 0x4002578700, 0x0, 0x0, 0x400476ef80, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:146 +0xa0 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).registerWithServer(0x4002578700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:301 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:205 +0x78 goroutine 164 [select]: net/http.(*persistConn).roundTrip(0x4004717440, 0x40023e5e00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004701e00, 0xf8, 0x10a8660, 0x400476e798) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004701e00, 0x1b6e900, 0xc03eba402109dde5, 0x1176fc08be6) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004701d00, 0x12db048, 0x1b6e900, 0xc03eba402109dde5, 0x1176fc08be6, 0x1be4a80, 0x0, 0x400011c6f8, 0x4004e22660, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004e22630, 0x4004701d00, 0xc03eba402109dde5, 0x1176fc08be6, 0x1be4a80, 0x400011c6f8, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004e22630, 0x4004701d00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004e024e0, 0x1f, 0x400476ed10, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/indexer.getWithAuth(0x4004e024e0, 0x1f, 0xe, 0x113846e, 0x11) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:3077 +0x68 github.com/couchbase/indexing/secondary/indexer.getGlobalTopology(0x4002561550, 0xe, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:2266 +0x78 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).updateNodeList(0x4002578700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:220 +0x3c created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:206 +0x98 goroutine 156 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x40001fa000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1090 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 157 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x40001fa000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1017 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 183 [IO wait]: internal/poll.runtime_pollWait(0xffff6c1e2870, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004e04418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004e04400, 0x4004ddc000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004e04400, 0x4004ddc000, 0x1000, 0x1000, 0x4002537be8, 0x7ff1a8, 0x4002537cb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001622c8, 0x4004ddc000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004717440, 0x4004ddc000, 0x1000, 0x1000, 0x60, 0x0, 0x4004e02401) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004dc6600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004dc6600, 0x1, 0x4004dde060, 0x0, 0x4002537e58, 0xd8c91c, 0x4004e024e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004717440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 186 [IO wait]: internal/poll.runtime_pollWait(0xffff6c1e25b8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004e04698, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004e04680, 0x4004de2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004e04680, 0x4004de2000, 0x1000, 0x1000, 0x4004de8be0, 0x469b10, 0x4004eaa780) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001622d8, 0x4004de2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004717680, 0x4004de2000, 0x1000, 0x1000, 0x4004de8cf8, 0x67867c, 0x4004dde180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004dc6780) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004dc6780, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004717680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 187 [select]: net/http.(*persistConn).writeLoop(0x4004717680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 169 [IO wait]: internal/poll.runtime_pollWait(0xffff6c1e26a0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000407218, 0x72, 0x0, 0x1, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000407200, 0x40046c1a81, 0x1, 0x1, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000407200, 0x40046c1a81, 0x1, 0x1, 0x4000437701, 0x4004e08700, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001622d0, 0x40046c1a81, 0x1, 0x1, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*connReader).backgroundRead(0x40046c1a70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:692 +0x50 created by net/http.(*connReader).startBackgroundRead /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:688 +0xc4 goroutine 172 [IO wait]: internal/poll.runtime_pollWait(0xffff6c1e24d0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004e04818, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004e04800, 0x4004e39000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004e04800, 0x4004e39000, 0x1000, 0x1000, 0x4004dd9be0, 0x469b10, 0x4004eaa600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011c740, 0x4004e39000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40047177a0, 0x4004e39000, 0x1000, 0x1000, 0x4004dd9cf8, 0x67867c, 0x4004e3c660) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004e36c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004e36c00, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40047177a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 173 [select]: net/http.(*persistConn).writeLoop(0x40047177a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 196 [select]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).wait(0x4002578700, 0x4004f32a30, 0x8, 0x9, 0x4002471e00, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:1807 +0xf4 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).GetTaskList(0x4002578700, 0x4004f32a30, 0x8, 0x9, 0x4002471e00, 0xadb658, 0x4004440a28, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:338 +0xd4 github.com/couchbase/cbauth/service.serviceAPI.GetTaskList.func1(0x4002471e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:99 +0x68 github.com/couchbase/cbauth/service.withTimeout(0x7530, 0x4004440b00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:46 +0x6c github.com/couchbase/cbauth/service.serviceAPI.GetTaskList(0x13007d8, 0x4002578700, 0x4004f32a30, 0x8, 0x9, 0x7530, 0x4004f3c0c0, 0x46b478, 0x4004c6e748) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:98 +0x94 reflect.Value.call(0x4004e36de0, 0x400011c7c0, 0x13, 0x112ec54, 0x4, 0x4004440f00, 0x3, 0x3, 0x2, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/reflect/value.go:476 +0x644 reflect.Value.Call(0x4004e36de0, 0x400011c7c0, 0x13, 0x4000435700, 0x3, 0x3, 0x40004356e8, 0x4bcf10, 0x4000099368) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/reflect/value.go:337 +0x84 net/rpc.(*service).call(0x4004e52040, 0x400012fa40, 0x4004759960, 0x4004759970, 0x4004e04a00, 0x4000121820, 0xfb9280, 0x40001c3720, 0x199, 0xee10c0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:377 +0x13c created by net/rpc.(*Server).ServeCodec /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:474 +0x2f8 goroutine 197 [select]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).wait(0x4002578700, 0x4004f32a60, 0x8, 0x9, 0x4002471d40, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:1807 +0xf4 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).GetCurrentTopology(0x4002578700, 0x4004f32a60, 0x8, 0x9, 0x4002471d40, 0xadb658, 0x4004440a28, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:390 +0xd4 github.com/couchbase/cbauth/service.serviceAPI.GetCurrentTopology.func1(0x4002471d40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:76 +0x68 github.com/couchbase/cbauth/service.withTimeout(0x7530, 0x4004440b00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:46 +0x6c github.com/couchbase/cbauth/service.serviceAPI.GetCurrentTopology(0x13007d8, 0x4002578700, 0x4004f32a60, 0x8, 0x9, 0x7530, 0x4000099400, 0x46b478, 0x4004c6e6c8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:75 +0x94 reflect.Value.call(0x4004e36d20, 0x400011c780, 0x13, 0x112ec54, 0x4, 0x4004440f00, 0x3, 0x3, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/reflect/value.go:476 +0x644 reflect.Value.Call(0x4004e36d20, 0x400011c780, 0x13, 0x400141bf00, 0x3, 0x3, 0x1d, 0x400004f6d8, 0x400476af00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/reflect/value.go:337 +0x84 net/rpc.(*service).call(0x4004e52040, 0x400012fa40, 0x4004759960, 0x4004759970, 0x4004e04900, 0x40001c3620, 0xfb91e0, 0x40001c3740, 0x199, 0xee1180, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:377 +0x13c created by net/rpc.(*Server).ServeCodec /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:474 +0x2f8 goroutine 158 [sleep]: time.Sleep(0x12a05f200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/plasma.RunMemQuotaTunerIteration(0x4001419fb0, 0xdf74d8, 0x4000182600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:211 +0x24c github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:247 +0xc0 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 goroutine 198 [select]: github.com/couchbase/cbauth/service.withTimeout.func1(0x7530, 0x4004f32a70, 0x4002471d40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:38 +0x78 created by github.com/couchbase/cbauth/service.withTimeout /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:36 +0xd8 goroutine 199 [select]: github.com/couchbase/cbauth/service.withTimeout.func1(0x7530, 0x4004f32a84, 0x4002471e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:38 +0x78 created by github.com/couchbase/cbauth/service.withTimeout /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:36 +0xd8 Initializing write barrier = 8000 2021-08-16T13:49:09.752+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:49:09.752+00:00 [Info] Setting ipv6=false 2021-08-16T13:49:09.763+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:49:09.769+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:49:09.769+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:49:09.769+00:00 [Info] Setting log level to Info 2021-08-16T13:49:09.770+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:49:09.772+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:49:09 audit: created new audit service 2021-08-16T13:49:09.781+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:49:09.781+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:49:09.781+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:49:09.781+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:49:09.781+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:49:09.792+00:00 [Info] security context initialized 2021-08-16T13:49:09.792+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:49:09.792+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:49:09.792+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:49:09.792+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:49:09.792+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:49:09.804+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:49:09.805+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:49:09.820+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:49:09.832+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:49:09.833+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:49:09.834+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:49:09.836+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:49:09.837+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:49:09.847+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:49:09.847+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:49:09.847+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:49:09.904+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:49:09.904+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 53 us 2021-08-16T13:49:09.905+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:49:09.906+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:49:09.907+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:49:09.911+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:49:09.911+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:49:09.911+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:49:09.911+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:49:09.911+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:49:09.911+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:09.911+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:09.914+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:49:09.914+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:49:09.914+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:49:09.914+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:09.914+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:49:09.914+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:49:09.914+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:49:09.914+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:49:09.914+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:49:09.914+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:49:09.914+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:49:09.919+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 4.257224ms 2021-08-16T13:49:09.919+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:49:09.920+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:49:09.920+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:09.920+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:49:09.920+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:49:09.920+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:49:09.920+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:49:09.931+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:49:09.931+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:49:09.931+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:49:09.931+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:49:09.931+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x40024c121f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x40024c121f pc=0xbd8ea4] goroutine 178 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004e529f0 sp=0x4004e529c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004e52a30 sp=0x4004e529f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x400457a680, 0x4004e64000, 0x11a9890, 0x4004e52b40, 0x400457a698, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004e52ab0 sp=0x4004e52a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x400457a680, 0x4004e64000, 0x11a9890, 0x4004e52b40, 0x400457a698, 0xffff8667bba8, 0x4004e52c18, 0x49) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004e52af0 sp=0x4004e52ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4000032b40, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004e52da0 sp=0x4004e52af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4000032b40, 0x40001bf180, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004e52e60 sp=0x4004e52da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004541530, 0x4004541540, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004e53200 sp=0x4004e52e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004541530, 0x4004541540, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004e538f0 sp=0x4004e53200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004e4a420, 0x4004e01180, 0x40001cfb80, 0x4004541510) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004e53fb0 sp=0x4004e538f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004e53fb0 sp=0x4004e53fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x40000e2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x400037e090, 0x4, 0x1136a87, 0xf, 0x4000119e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 18 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x4000196108) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 19 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x4000196120) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 20 [IO wait]: internal/poll.runtime_pollWait(0xffff86643758, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0100, 0x4000141000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0100, 0x4000141000, 0x1000, 0x1000, 0x1311860, 0x47b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011e010, 0x4000141000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x400010e180, 0x40001f0001, 0xdff, 0xdff, 0x400009f8e8, 0x6902d8, 0x4000115948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400011a030, 0x40001f0001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40001a0160, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40001a0160, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40001a0160, 0xf5a960, 0x4000180920, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4000180900, 0x400018a580, 0x4000115b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400019a690, 0x12f8f98, 0x4000180900, 0x400018cfe7, 0x16, 0x40001f2120, 0x400009fc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400019a690, 0x12f8f98, 0x4000180900, 0x400019a690, 0x400018c7f8, 0x400018c800, 0x4000190780, 0x400018a4a0, 0xee0980, 0x40001f2120, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400019a690, 0x12f8f98, 0x4000180900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x40001827e0, 0x400009ff78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400009df78, 0x40001827e0, 0x12db8a8, 0x400000e570, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x40001827e0, 0x40001a4640, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x40001827e0, 0x40001a4640) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 22 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 23 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 24 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 25 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 26 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 34 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400010c300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 27 [syscall]: syscall.Syscall(0x3f, 0x0, 0x4000055fa0, 0x4, 0x4000055e68, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x4000055fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x4000055fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x4000055fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 32 [select]: net/http.(*persistConn).writeLoop(0x400000d0e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 12 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000186c30, 0x12c18bc, 0x1, 0x40000a1e58, 0x40000387e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x4000159ec8, 0x40000387e0, 0x0, 0x4000038a78) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x4714f4, 0x4000038a20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40000a1fa0, 0x4000036000, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4000080000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 29 [select]: net/http.(*persistConn).readLoop(0x400000c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 30 [select]: net/http.(*persistConn).writeLoop(0x400000c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 13 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4000080000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 51 [IO wait]: internal/poll.runtime_pollWait(0xffff86643670, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0180, 0x4000140000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0180, 0x4000140000, 0x1000, 0x1000, 0x40013a6a88, 0x68f7a4, 0x40001a0e10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000100e0, 0x4000140000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c5a0, 0x4000140000, 0x1000, 0x1000, 0xf12180, 0x400008c310, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400010e4e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400010e4e0, 0xa, 0x1008880, 0x400008c2d0, 0x199, 0x1000, 0x40013a6b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400010e4e0, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000183350) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000183350, 0x40013aa000, 0xe00, 0xe00, 0x40013a6cc8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000081740, 0x40013aa000, 0xe00, 0xe00, 0x40013a6e08, 0x40bb94, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000081740, 0x40013aa000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000081780, 0x40013aa000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40001a0dc0, 0x0, 0x40013a6eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40001a0dc0, 0x0, 0x0, 0x40013a6f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40001a0dc0, 0xee0d40, 0x400008c2d0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40001ba7e0, 0x4000082d20, 0x4000038f60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 16 [IO wait]: internal/poll.runtime_pollWait(0xffff86643588, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0280, 0x40000fe000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0280, 0x40000fe000, 0x1000, 0x1000, 0x40013a4be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010460, 0x40000fe000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000cd80, 0x40000fe000, 0x1000, 0x1000, 0x40013a4cf8, 0x67867c, 0x4000038f00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000082ba0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000082ba0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000cd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 50 [select]: net/http.(*persistConn).writeLoop(0x400000cd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 64 [IO wait]: internal/poll.runtime_pollWait(0xffff86643018, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1018, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f1000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40046ff278, 0x203020202020203, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40046ff278, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x40001711c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 118 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x4002508f00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 79 [IO wait]: internal/poll.runtime_pollWait(0xffff86642c78, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000191218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000191200, 0x40013de000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000191200, 0x40013de000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010320, 0x40013de000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000d0e0, 0x40013de000, 0x1000, 0x1000, 0x40001c1800, 0x200000003, 0x40001c1800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002538d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002538d80, 0x400011200a, 0x40023878c8, 0x4d3b10, 0x40000820c0, 0x4000112000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4002538d80, 0x0, 0x1, 0x0, 0x40000820d8, 0x4000112000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400012f590) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400012f590, 0x40013e0000, 0x1000, 0x1000, 0x40023879b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000081900, 0x40013e0000, 0x1000, 0x1000, 0x40000ce418, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000081900, 0x40013e0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000081940, 0x40013e0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4002387dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002387dd0, 0xa, 0x40013e0201, 0x1, 0xffff8667cd30, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4002387dd0, 0x40013e020a, 0x1, 0x400018cc50, 0x0, 0x40013e02e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4002387dd0, 0x400014350a, 0x400018cc50, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40001998a0, 0x4004460348, 0x17, 0x4002387f20, 0x400242c7f8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x40001998a0, 0x113075a, 0x7, 0x400242c7f8, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x4000199860) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 35 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000170dc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 109 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x400007f260) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 37 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x4002520000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 47 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x4000097c80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 46 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x400037b380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 40 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x4002520000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 41 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x4002520000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 42 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x4002520000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 45 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40000819c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 58 [select]: net/http.(*persistConn).readLoop(0x400037de60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 59 [select]: net/http.(*persistConn).writeLoop(0x400037de60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 48 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x4000190000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 49 [IO wait]: internal/poll.runtime_pollWait(0xffff86642e48, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000190598, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000190580, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000190580, 0x4714f4, 0x4000157698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x400242c288, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x400242c288, 0x12ed9d0, 0x400242c288, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x4000190100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 66 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4000199040, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 67 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4000199040, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 68 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x4000199040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 69 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x40001113b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 85 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40013e4540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 73 [IO wait]: internal/poll.runtime_pollWait(0xffff86642d60, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000190718, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000190700, 0x4000375000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000190700, 0x4000375000, 0x1000, 0x1000, 0x40013a7be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4004476068, 0x4000375000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c6c0, 0x4000375000, 0x1000, 0x1000, 0x40013a7cf8, 0x67867c, 0x4000039140) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400037bbc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x400037bbc0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000c6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 72 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40000e01c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 74 [select]: net/http.(*persistConn).writeLoop(0x400000c6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 108 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x400007f260) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 80 [IO wait]: internal/poll.runtime_pollWait(0xffff86642f30, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f1400, 0x4002443000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f1400, 0x4002443000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011e480, 0x4002443000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400037de60, 0x4002443000, 0x1000, 0x1000, 0x40024b5080, 0x200000003, 0x40024b5080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000083ce0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000083ce0, 0x400011200a, 0x40013c18c8, 0x4d3b10, 0x40000820c0, 0x4000112000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000083ce0, 0x0, 0x1, 0x0, 0x40000820d8, 0x4000112000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400037f500) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400037f500, 0x40000f8000, 0x1000, 0x1000, 0x40013c19b8, 0x0, 0x40013c1988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004462580, 0x40000f8000, 0x1000, 0x1000, 0x4000103200, 0x3, 0x40013c1be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004462580, 0x40000f8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40044625c0, 0x40000f8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40013c1dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40013c1dd0, 0xa, 0x40000f8201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40013c1dd0, 0x40000f820a, 0x1, 0x400242f9cb, 0x0, 0x40000f8263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40013c1dd0, 0x400012230a, 0x400242f9cb, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40001998a0, 0x400445c6c0, 0x24, 0x40013c1f20, 0x400242c7e0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x40001998a0, 0x113075a, 0x7, 0x400242c7e0, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x4000199860) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 114 [select]: net/http.(*persistConn).writeLoop(0x400000dd40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 31 [select]: net/http.(*persistConn).readLoop(0x400000d0e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 63 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400018f7c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 116 [select]: net/http.(*persistConn).readLoop(0x40044d25a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 101 [IO wait]: internal/poll.runtime_pollWait(0xffff86642aa8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000191498, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000191480, 0x40013e1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000191480, 0x40013e1000, 0x1000, 0x1000, 0x4002385688, 0x69f178, 0x40023856f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010378, 0x40013e1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000dd40, 0x40013e1000, 0x1000, 0x1000, 0x5, 0x400240c500, 0x40023857c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002538e40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002538e40, 0x68f90a, 0x40001f9340, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4002538e40, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400249cc90) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400249cc90, 0x40024e8000, 0x1000, 0x1000, 0x4002385998, 0x44e884, 0x4002385948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400018e840, 0x40024e8000, 0x1000, 0x1000, 0x40000ce418, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400018e840, 0x40024e8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400018e880, 0x40024e8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4002385d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002385d80, 0xa, 0x40024e8201, 0x1, 0xffff5c76c970, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4002385d80, 0x40024e820a, 0x1, 0x400451a6d0, 0x0, 0x40024e8259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4002385d80, 0x40023fdb0a, 0x400451a6d0, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40001998a0, 0x4002422760, 0x1f, 0x4002385ee0, 0x4002421c80, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x40001998a0, 0x113075a, 0x7, 0x40024a35e0, 0xd, 0x4002421c80, 0x0, 0x2a, 0x40001557a8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x4000199860, 0x40024a35e0, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 115 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000170400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 33 [select]: net/http.(*persistConn).readLoop(0x400000dd40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 117 [select]: net/http.(*persistConn).writeLoop(0x40044d25a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 65 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x40013e4c00, 0x40013b48c0, 0x40013b4930, 0x40045e4240, 0x12e8f90, 0x4002390460, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x40046e51b4, 0x5, 0x40001711c0, 0x12eca70, 0x4000170e80, 0x130d478, 0x400035e5b0, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40000e8460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40000e8460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 119 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x4002508f00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 120 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x40013e4c00, 0x40045e4240) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 121 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x4002390460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 122 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x4002390460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 123 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x400243ebd0, 0x40045a7500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 124 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x40001703c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 125 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x4002508fa0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 126 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x4002508fa0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 127 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x4000010818) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 128 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x4002508fa0, 0x40045a7560) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 129 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000186c30, 0x1144559, 0x1b, 0x400009be38, 0x40045a7680, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x40045e4d60, 0x40045a7680, 0xf1f700, 0x40003ce4b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400009bf80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004d025a0, 0x40045e4d60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 130 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 131 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x4000150c30, 0xd00007, 0x4004d04068, 0x5, 0x40045e4d70, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 132 [IO wait]: internal/poll.runtime_pollWait(0xffff866429c0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4002523118, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4002523100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4002523100, 0xffff866faa68, 0x30, 0x40045b3b30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40045e2660, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40045e2660, 0x40045b3b30, 0x11ae500, 0x4002520860, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40025207e0, 0x12ed9d0, 0x40045e2660, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40025207e0, 0x12ed9d0, 0x40045e2660, 0x40000e2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 133 [IO wait]: internal/poll.runtime_pollWait(0xffff866428d8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4002523218, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4002523200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4002523200, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40045e2678, 0xffff866faa68, 0xffff5c76e730, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40045e2678, 0x665e1c, 0x0, 0xfe5e60, 0x4004d135c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x40045e2690, 0x40045b3bc0, 0x11ae500, 0x4002520940, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40025208c0, 0x12ec710, 0x40045e2690, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40025208c0, 0x12ec710, 0x40045e2690, 0x40000e2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 148 [IO wait]: internal/poll.runtime_pollWait(0xffff5c9d79c8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f1180, 0x4004753000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f1180, 0x4004753000, 0x1000, 0x1000, 0x4004d1abe0, 0x469b10, 0x40045b7c80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011e450, 0x4004753000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004cf6900, 0x4004753000, 0x1000, 0x1000, 0x4004d1acf8, 0x67867c, 0x400475a060) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004718540) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004718540, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004cf6900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 146 [IO wait]: internal/poll.runtime_pollWait(0xffff86642b90, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000191998, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000191980, 0x40045b4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000191980, 0x40045b4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010648, 0x40045b4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40044d25a0, 0x40045b4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40025394a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40025394a0, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40025394a0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400471f0b0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400471f0b0, 0x400471cc00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40001713c0, 0x400471cc00, 0x200, 0x200, 0x115b899, 0x22, 0x1156d69) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40001713c0, 0x400471cc00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000171400, 0x400471cc00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004720840, 0x0, 0x400036ceb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004720840, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004720840, 0xee0d40, 0x400019a960, 0x12db068, 0x4000171400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004740510, 0x40047184e0, 0x400017de60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 149 [select]: net/http.(*persistConn).writeLoop(0x4004cf6900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 107 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x400007f260) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 106 [semacquire]: sync.runtime_Semacquire(0x4004e4a428) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004e4a420) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x40001cfb80, 0x4000182960, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffca114969, 0x29, 0xffffca1149cb, 0x25, 0x400452bec0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x40046e5ea0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffca114969, 0x29, 0xffffca1149cb, 0x25, 0x400452bec0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x40046e5ea0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004e53190, 0x4004e52e60, 0x4000182960, 0x4000165440, 0x4000170dc0, 0x0, 0xffff866faf40, 0x400009eda8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x40000e2000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x40046e5ea0, 0xb, 0x40046e5e98, 0x6, 0x40046e5eb0, 0xd, 0x4004700960, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x40000e2000, 0x4000370000, 0x4000096d50) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x40000e2000, 0x400006dbc0, 0x2, 0x2, 0x400006dbd0, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 110 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x400007f260) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 111 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4004c5c640) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 112 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40047e2f20, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 113 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40047e2f20, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 162 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x40047e2f20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 163 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x400007f260) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 164 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000186c30, 0x1144559, 0x1b, 0x4002485e38, 0x4004dd61e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004679980, 0x4004dd61e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4002485f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40046774a0, 0x4004679980) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 169 [IO wait]: internal/poll.runtime_pollWait(0xffff5c9d78e0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004dd0498, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004dd0480, 0x4004dec000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004dd0480, 0x4004dec000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4004476428, 0x4004dec000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40044d3560, 0x4004dec000, 0x1000, 0x1000, 0x20, 0x30, 0x20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004677f80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004677f80, 0x100a, 0x0, 0x0, 0x0, 0x0, 0x4000153438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004677f80, 0x412b8c, 0x4000153478, 0x655784, 0x40001534a8, 0x40b2f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004c332f0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004c332f0, 0x4004c1b600, 0x200, 0x200, 0x4, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004c5c780, 0x4004c1b600, 0x200, 0x200, 0x4004647ec7, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004c5c780, 0x4004c1b600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004c5c7c0, 0x4004c1b600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004dd8420, 0x0, 0x40001536b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004dd8420, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004dd8420, 0xee0d40, 0x4000014ff0, 0x12db068, 0x4004c5c7c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40047fbb00, 0x4004dee120, 0x4004dd6d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 167 [select]: net/http.(*persistConn).readLoop(0x40044d3560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 168 [select]: net/http.(*persistConn).writeLoop(0x40044d3560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 87 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4004c5c680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 88 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x40047e2dc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 89 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x40047e2dc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 90 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x40047e2dc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 91 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x40047e2dc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 92 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004572b00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 93 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x4004572b00, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 94 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x40000e2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 95 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x40000e2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 96 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x40000e2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 97 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:49:15.081+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:49:15.081+00:00 [Info] Setting ipv6=false 2021-08-16T13:49:15.087+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:49:15.099+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:49:15.099+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:49:15.099+00:00 [Info] Setting log level to Info 2021-08-16T13:49:15.099+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:49:15.103+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:49:15 audit: created new audit service 2021-08-16T13:49:15.109+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:49:15.109+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:49:15.109+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:49:15.109+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:49:15.109+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:49:15.119+00:00 [Info] security context initialized 2021-08-16T13:49:15.119+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:49:15.120+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:49:15.120+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:49:15.120+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:49:15.120+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:49:15.140+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:49:15.146+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:49:15.144+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:49:15.163+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:49:15.163+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:49:15.163+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:49:15.165+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:49:15.166+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:49:15.167+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:49:15.167+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:49:15.167+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:49:15.220+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 55 us 2021-08-16T13:49:15.221+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:49:15.220+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:49:15.223+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:49:15.223+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:49:15.224+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:49:15.224+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:49:15.224+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:49:15.225+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:49:15.225+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:49:15.227+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:15.227+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:15.229+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:49:15.229+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:49:15.229+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:49:15.229+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:15.229+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:49:15.229+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:49:15.229+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:49:15.229+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:49:15.229+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:49:15.229+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:49:15.229+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:49:15.234+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 4.926372ms 2021-08-16T13:49:15.234+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:49:15.235+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:49:15.235+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:15.235+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:49:15.235+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:49:15.236+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:49:15.236+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:49:15.246+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:49:15.246+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:49:15.247+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:49:15.247+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:49:15.247+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x4004c0cb5f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x4004c0cb5f pc=0xbd8ea4] goroutine 192 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004be69f0 sp=0x4004be69c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004be6a30 sp=0x4004be69f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x400442ad00, 0x4004e0b200, 0x11a9890, 0x4004be6b40, 0x400442ad18, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004be6ab0 sp=0x4004be6a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x400442ad00, 0x4004e0b200, 0x11a9890, 0x4004be6b40, 0x400442ad18, 0xffff687afa98, 0x4004be6c18, 0x49) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004be6af0 sp=0x4004be6ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4004e043c0, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004be6da0 sp=0x4004be6af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4004e043c0, 0x4004418cb0, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004be6e60 sp=0x4004be6da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x400016ff00, 0x400016ff10, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004be7200 sp=0x4004be6e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x400016ff00, 0x400016ff10, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004be78f0 sp=0x4004be7200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004ea8f90, 0x4004eae700, 0x4004e00b00, 0x400016fee0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004be7fb0 sp=0x4004be78f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004be7fb0 sp=0x4004be7fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x4000240000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x4000191470, 0x4, 0x1136a87, 0xf, 0x40000f9e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 18 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x40001b20d8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 19 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x40001b20f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 20 [IO wait]: internal/poll.runtime_pollWait(0xffff68732e88, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000210118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000210100, 0x400022b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000210100, 0x400022b000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000206020, 0x400022b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x40002002a0, 0x40000fe001, 0xdff, 0xdff, 0x400009f8e8, 0x6902d8, 0x40000f5948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x4000212030, 0x40000fe001, 0xdff, 0xdff, 0x40000f59f8, 0x6f1968, 0x40000f5938) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40000ce580, 0xa, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40000ce580, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40000ce580, 0xf5a960, 0x4000082440, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4000082420, 0x40000462e0, 0x40000f5b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400008c190, 0x12f8f98, 0x4000082420, 0x4000036c77, 0x16, 0x400000d320, 0x400012ec01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400008c190, 0x12f8f98, 0x4000082420, 0x400008c190, 0x4000036488, 0x4000036490, 0x40000f0080, 0x4000046200, 0xee0980, 0x400000d320, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400008c190, 0x12f8f98, 0x4000082420) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x40001903f0, 0x400012ef78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x40024f3f78, 0x40001903f0, 0x12db8a8, 0x4000212000, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x40001903f0, 0x40001a85a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x40001903f0, 0x40001a85a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 21 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 22 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 23 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 24 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 25 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 34 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x4000232100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 26 [syscall]: syscall.Syscall(0x3f, 0x0, 0x4000055fa0, 0x4, 0x4000055e68, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x4000055fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x4000055fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x4000055fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 8 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400019a380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 42 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001926f0, 0x12c18bc, 0x1, 0x4000125e58, 0x4000216720, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x400141a6c8, 0x4000216720, 0x400141a6c8, 0x679704) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000125fa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x400020e180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 40 [select]: net/http.(*persistConn).readLoop(0x400000c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 41 [select]: net/http.(*persistConn).writeLoop(0x400000c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 43 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x400020e180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 72 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x400440a500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 46 [IO wait]: internal/poll.runtime_pollWait(0xffff68732cb8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000120318, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000120300, 0x400025c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000120300, 0x400025c000, 0x1000, 0x1000, 0x400006abe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000102f8, 0x400025c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000cb40, 0x400025c000, 0x1000, 0x1000, 0x400006acf8, 0x67867c, 0x4000216e40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000200fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000200fc0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000cb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 47 [select]: net/http.(*persistConn).writeLoop(0x400000cb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 28 [IO wait]: internal/poll.runtime_pollWait(0xffff68732da0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000120198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000120180, 0x400022a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000120180, 0x400022a000, 0x1000, 0x1000, 0x4000130a88, 0x68f7a4, 0x40001a4470) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000100b8, 0x400022a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c5a0, 0x400022a000, 0x1000, 0x1000, 0xf12180, 0x400015c0e0, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000200a80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000200a80, 0xa, 0x1008880, 0x400015c0a0, 0x199, 0xdd4, 0x4000130b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000200a80, 0x2, 0x0, 0x9, 0x56b, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4002368180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4002368180, 0x4000014000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40000805c0, 0x4000014000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40000805c0, 0x4000014000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40000816c0, 0x4000014000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40001a4420, 0x0, 0x4000130eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40001a4420, 0x0, 0x0, 0x4000130f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40001a4420, 0xee0d40, 0x400015c0a0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40000de120, 0x40003f6a80, 0x40001c0180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 33 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400011e380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 118 [IO wait]: internal/poll.runtime_pollWait(0xffff687d1de8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000121698, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000121680, 0x4004b79000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000121680, 0x4004b79000, 0x1000, 0x1000, 0x4004c3dbe0, 0x469b10, 0x400460c000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000206298, 0x4004b79000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000d9e0, 0x4004b79000, 0x1000, 0x1000, 0x4004c3dcf8, 0x67867c, 0x4004b7e000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004543080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004543080, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000d9e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 14 [IO wait]: internal/poll.runtime_pollWait(0xffff68732490, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001c8298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40001c8280, 0x400247e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40001c8280, 0x400247e000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000194138, 0x400247e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004410240, 0x400247e000, 0x1000, 0x1000, 0x4000182780, 0x200000003, 0x4000182780) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003f73e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003f73e0, 0x400001800a, 0x40001238c8, 0x4d3b10, 0x40000820c0, 0x4000018000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003f73e0, 0x0, 0x1, 0x0, 0x40000820d8, 0x4000018000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4002438ba0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4002438ba0, 0x4004438000, 0x1000, 0x1000, 0x40001239b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400020e440, 0x4004438000, 0x1000, 0x1000, 0x4000083438, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400020e440, 0x4004438000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400020e480, 0x4004438000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000123dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000123dd0, 0xa, 0x4004438201, 0x1, 0xffff6872ef70, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000123dd0, 0x400443820a, 0x1, 0x400020d640, 0x0, 0x40044382e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000123dd0, 0x400019d50a, 0x400020d640, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400019d2f0, 0x40001701f8, 0x17, 0x4000123f20, 0x400247c000, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x400019d2f0, 0x113075a, 0x7, 0x400247c000, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x400019d2b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 107 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400011e7c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 12 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x400251a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 11 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x400251a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 10 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x400251a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 13 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x400251a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 105 [select]: net/http.(*persistConn).readLoop(0x400021e6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 82 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x4000082de0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 83 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x40002036e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 84 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x4000120100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 85 [IO wait]: internal/poll.runtime_pollWait(0xffff68732578, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000120398, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000120380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000120380, 0x680da0, 0x4000052698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40001b39f8, 0x40001c0a80, 0x40003f7800, 0x4000102688) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40001b39f8, 0x12ed9d0, 0x40001b39f8, 0x58ec48, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x4000120280) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 15 [IO wait]: internal/poll.runtime_pollWait(0xffff687323a8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0718, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0700, 0x4000016000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0700, 0x4000016000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001160f8, 0x4000016000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001f6c60, 0x4000016000, 0x1000, 0x1000, 0x4000102900, 0x200000003, 0x4000102900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40024767e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40024767e0, 0x400001800a, 0x400257f8c8, 0x4d3b10, 0x40000820c0, 0x4000018000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40024767e0, 0x0, 0x1, 0x7f, 0x40000820d8, 0x4000018000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000190d50) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000190d50, 0x40024f5000, 0x1000, 0x1000, 0x400257f9b8, 0x0, 0x400257f988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400011e680, 0x40024f5000, 0x1000, 0x1000, 0x4000001b00, 0x3, 0x400257fbe0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400011e680, 0x40024f5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400011e6c0, 0x40024f5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400257fdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400257fdd0, 0xa, 0x40024f5201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400257fdd0, 0x40024f520a, 0x1, 0x400046bd2b, 0x0, 0x40024f5263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400257fdd0, 0x400004670a, 0x400046bd2b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400019d2f0, 0x400015e6f0, 0x24, 0x400257ff20, 0x4002573938, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x400019d2f0, 0x113075a, 0x7, 0x4002573938, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x400019d2b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 86 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400022cd00, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 87 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400022cd00, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 88 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x400022cd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 89 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x4000097ce0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 70 [select]: net/http.(*persistConn).readLoop(0x4004410240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 71 [select]: net/http.(*persistConn).writeLoop(0x4004410240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 90 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40003d45b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 106 [select]: net/http.(*persistConn).writeLoop(0x400021e6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 99 [select]: net/http.(*persistConn).readLoop(0x40001f6c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 100 [select]: net/http.(*persistConn).writeLoop(0x40001f6c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 103 [IO wait]: internal/poll.runtime_pollWait(0xffff687322c0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0a98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0a80, 0x400001e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0a80, 0x400001e000, 0x1000, 0x1000, 0x4004483be0, 0x469b10, 0x4000103680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000116130, 0x400001e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001f6d80, 0x400001e000, 0x1000, 0x1000, 0x4004483cf8, 0x67867c, 0x40001c1440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002476cc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4002476cc0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40001f6d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 104 [select]: net/http.(*persistConn).writeLoop(0x40001f6d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 93 [IO wait]: internal/poll.runtime_pollWait(0xffff687321d8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000120718, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000120700, 0x40024fa000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000120700, 0x40024fa000, 0x1000, 0x1000, 0x4002579688, 0x69f178, 0x40025796f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000206138, 0x40024fa000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400021e7e0, 0x40024fa000, 0x1000, 0x1000, 0x5, 0x400444a000, 0x40025797c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000083680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000083680, 0x68f90a, 0x40025798a8, 0x412bc0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000083680, 0x0, 0x4002579868, 0x41a0a8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4002439020) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4002439020, 0x4004445000, 0x1000, 0x1000, 0x4002579998, 0x44e884, 0x40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400020e680, 0x4004445000, 0x1000, 0x1000, 0x40001770d8, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400020e680, 0x4004445000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400020e6c0, 0x4004445000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4002579d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002579d80, 0xa, 0x4004445201, 0x1, 0xffff6870eb28, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4002579d80, 0x400444520a, 0x1, 0x4004596620, 0x0, 0x4004445259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4002579d80, 0x40023eed0a, 0x4004596620, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400019d2f0, 0x40023c3240, 0x1f, 0x4002579ee0, 0x40001b3b00, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x400019d2f0, 0x113075a, 0x7, 0x400046bf20, 0xd, 0x40001b3b00, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x400019d2b0, 0x400046bf20, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 60 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x4000081c00, 0x400461b940) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 96 [select]: net/http.(*persistConn).readLoop(0x400021e7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 97 [select]: net/http.(*persistConn).writeLoop(0x400021e7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 108 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400019b780) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 131 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x4000081c00, 0x400007ed90, 0x400007ee00, 0x400461b940, 0x12e8f90, 0x40000e8a80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x40045f0334, 0x5, 0x400019bfc0, 0x12eca70, 0x400019be80, 0x130d478, 0x40000e01c0, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40000e8d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40000e8d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 130 [IO wait]: internal/poll.runtime_pollWait(0xffff687320f0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1b18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f1b00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f1b00, 0x58ec48, 0x400459f260, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40045dfb90, 0x4714f4, 0x400459f260, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40045dfb90, 0x4004597f00, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x400019bfc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 113 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400019bb40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 73 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x400440a500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 132 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40000e8a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 133 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40000e8a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 134 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x4000218e10, 0x400459f860) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 135 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x400011e780) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 136 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x4000013040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 137 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x4000013040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 138 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x40001166d8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 139 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x4000013040, 0x400459f8c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 140 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001926f0, 0x1144559, 0x1b, 0x400024de38, 0x400459f9e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x40045a9410, 0x400459f9e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400024df80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40045ea720, 0x40045a9410) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 141 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 142 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400019ef00, 0xd00007, 0x40045f1b90, 0x5, 0x40045a9420, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 143 [IO wait]: internal/poll.runtime_pollWait(0xffff68732748, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1c98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f1c80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f1c80, 0xffff9067aa68, 0x30, 0x400457d110) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004d1c090, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004d1c090, 0x400457d110, 0x11ae500, 0x40000e8e80, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40000e8e00, 0x12ed9d0, 0x4004d1c090, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40000e8e00, 0x12ed9d0, 0x4004d1c090, 0x4000240000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 144 [IO wait]: internal/poll.runtime_pollWait(0xffff68732008, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1d98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f1d80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f1d80, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004d1c0a8, 0xffff9067a108, 0xffff687a1210, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004d1c0a8, 0x665e1c, 0x0, 0xfe5e60, 0x4004d285c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004d1c0c0, 0x40045f6cf0, 0x11ae500, 0x40000e8f60, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40000e8ee0, 0x12ec710, 0x4004d1c0c0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40000e8ee0, 0x12ec710, 0x4004d1c0c0, 0x4000240000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 119 [select]: net/http.(*persistConn).writeLoop(0x400000d9e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 117 [IO wait]: internal/poll.runtime_pollWait(0xffff68732660, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000120598, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000120580, 0x40000d1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000120580, 0x40000d1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000116140, 0x40000d1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400021e6c0, 0x40000d1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002476ea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002476ea0, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4002476ea0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400457d1d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400457d1d0, 0x4004aef000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400011fbc0, 0x4004aef000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400011fbc0, 0x4004aef000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400011fc00, 0x4004aef000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004b76840, 0x400459f900, 0x40044406b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004b76840, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004b76840, 0xee0d40, 0x400015ca50, 0x12db068, 0x400011fc00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004531b90, 0x4004542fc0, 0x400452bec0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 165 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40000e8fc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 164 [semacquire]: sync.runtime_Semacquire(0x4004ea8f98) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004ea8f90) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x4004e00b00, 0x400010e630, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffc8d7c969, 0x29, 0xffffc8d7c9cb, 0x25, 0x4004e78d80, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x40045f1020, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffc8d7c969, 0x29, 0xffffc8d7c9cb, 0x25, 0x4004e78d80, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x40045f1020, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004be7190, 0x4004be6e60, 0x400010e630, 0x400047dd40, 0x400019a380, 0x0, 0xbf, 0x40024f0da8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x4000240000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x40045f1020, 0xb, 0x40045f1018, 0x6, 0x40045f1030, 0xd, 0x40045e1e00, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x4000240000, 0x40003f0000, 0x4000202570) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x4000240000, 0x4000204b90, 0x2, 0x2, 0x4000204ba0, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 76 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x400017cf20, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 75 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x400017cf20, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 166 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40000e8fc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 167 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40000e8fc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 168 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40000e8fc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 169 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x40045f8780) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 77 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x400017cf20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 78 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40000e8fc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 79 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001926f0, 0x1144559, 0x1b, 0x4000129e38, 0x4004d4a300, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x400016f580, 0x4004d4a300, 0x4004410240, 0x4001415640) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x12c2e00, 0x400016a501) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000129f80, 0x12c2e00, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40045eaa20, 0x400016f580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 180 [IO wait]: internal/poll.runtime_pollWait(0xffff687d1d00, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001c8d98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40001c8d80, 0x4004e80000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40001c8d80, 0x4004e80000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001942a0, 0x4004e80000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004410900, 0x4004e80000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004c11320) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004c11320, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004c11320, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004e745d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004e745d0, 0x4004c03a00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400017aac0, 0x4004c03a00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400017aac0, 0x4004c03a00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400017ab00, 0x4004c03a00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004e72c60, 0x0, 0x4004d296b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004e72c60, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004e72c60, 0xee0d40, 0x400008ca00, 0x12db068, 0x400017ab00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40000dfdd0, 0x4004c116e0, 0x4004e786c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 178 [select]: net/http.(*persistConn).readLoop(0x4004410900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 179 [select]: net/http.(*persistConn).writeLoop(0x4004410900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 172 [IO wait]: internal/poll.runtime_pollWait(0xffff687d1c18, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004d4c098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004d4c080, 0x40045f5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004d4c080, 0x40045f5000, 0x1000, 0x1000, 0x400012bbe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000116750, 0x40045f5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40045fe900, 0x40045f5000, 0x1000, 0x1000, 0x400012bcf8, 0x67867c, 0x4004d4aa20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40045ead20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40045ead20, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40045fe900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 173 [select]: net/http.(*persistConn).writeLoop(0x40045fe900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 181 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40045f87c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 182 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x4004d38370) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 183 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x4004d38370) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 184 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x4004d38370) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 185 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x4004d38370) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 186 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004eac100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 187 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x4004eac100, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 188 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x4000240000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 189 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x4000240000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 190 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x4000240000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 191 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:49:20.405+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:49:20.405+00:00 [Info] Setting ipv6=false 2021-08-16T13:49:20.411+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:49:20.413+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:49:20.413+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:49:20.413+00:00 [Info] Setting log level to Info 2021-08-16T13:49:20.413+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:49:20.415+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:49:20 audit: created new audit service 2021-08-16T13:49:20.423+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:49:20.423+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:49:20.424+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:49:20.424+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:49:20.424+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:49:20.434+00:00 [Info] security context initialized 2021-08-16T13:49:20.434+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:49:20.434+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:49:20.435+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:49:20.435+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:49:20.435+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:49:20.447+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:49:20.448+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:49:20.456+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:49:20.472+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:49:20.473+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:49:20.474+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:49:20.474+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:49:20.474+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:49:20.475+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:49:20.478+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:49:20.482+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:49:20.525+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:49:20.525+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 153 us 2021-08-16T13:49:20.526+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:49:20.528+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:49:20.528+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:49:20.533+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:49:20.533+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:49:20.533+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:49:20.533+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:49:20.533+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:49:20.533+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:20.533+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:20.535+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:49:20.535+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:49:20.535+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:49:20.535+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:20.535+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:49:20.535+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:49:20.535+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:49:20.536+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:49:20.536+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:49:20.536+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:49:20.536+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:49:20.540+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 4.168946ms 2021-08-16T13:49:20.540+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:49:20.542+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:49:20.542+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:20.542+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:49:20.542+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:49:20.542+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:49:20.542+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... unexpected fault address 0x4004c686df fatal error: fault 2021-08-16T13:49:20.553+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:49:20.553+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:49:20.553+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:49:20.553+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:49:20.553+00:00 [Info] : Recover shard shard1 from metadata completed. [signal SIGBUS: bus error code=0x1 addr=0x4004c686df pc=0xbd8ea4] goroutine 203 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004d8a9f0 sp=0x4004d8a9c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004d8aa30 sp=0x4004d8a9f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x4004dd6340, 0x4004beac00, 0x11a9890, 0x4004d8ab40, 0x4004dd6358, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004d8aab0 sp=0x4004d8aa40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x4004dd6340, 0x4004beac00, 0x11a9890, 0x4004d8ab40, 0x4004dd6358, 0xc18f8c, 0x4004d8ac18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004d8aaf0 sp=0x4004d8aab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x40000328c0, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004d8ada0 sp=0x4004d8aaf0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x40000328c0, 0x40002552d0, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004d8ae60 sp=0x4004d8ada0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004dca7a0, 0x4004dca7b0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004d8b200 sp=0x4004d8ae60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004dca7a0, 0x4004dca7b0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004d8b8f0 sp=0x4004d8b200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004df0770, 0x4004c6bc00, 0x40002e8680, 0x4004dca780) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004d8bfb0 sp=0x4004d8b8f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004d8bfb0 sp=0x4004d8bfb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x4000202700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x4000283650, 0x4, 0x1136a87, 0xf, 0x400028be68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 34 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400028c108) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 35 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400028c120) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 36 [IO wait]: internal/poll.runtime_pollWait(0xffffbc21e1f8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011e118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011e100, 0x4000124000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011e100, 0x4000124000, 0x1000, 0x1000, 0x1311860, 0x47b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000290098, 0x4000124000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x400010e300, 0x4000125001, 0xdff, 0xdff, 0x40001a18e8, 0x6902d8, 0x4000287948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x4000120018, 0x4000125001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x4000134000, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4000134000, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4000134000, 0xf5a960, 0x400010e3e0, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x400010e3c0, 0x4000118160, 0x4000287b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400012e000, 0x12f8f98, 0x400010e3c0, 0x400011c897, 0x16, 0x400013a120, 0x40001a1c01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400012e000, 0x12f8f98, 0x400010e3c0, 0x400012e000, 0x400011c0a0, 0x400011c0b0, 0x400011e180, 0x4000118080, 0xee0980, 0x400013a120, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400012e000, 0x12f8f98, 0x400010e3c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x40002824b0, 0x40001a1f78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x4000091f78, 0x40002824b0, 0x12db8a8, 0x4000120000, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x40002824b0, 0x40002b6280, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x40002824b0, 0x40002b6280) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 20 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400018c300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 37 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 38 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 39 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 40 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 41 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 42 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40002d9fa0, 0x4, 0x40002d9e68, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40002d9fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x40001c2000, 0x40002d9fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x40001c0000, 0x40002d9fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 66 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001e1c80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 55 [select]: net/http.(*persistConn).readLoop(0x400000c120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 56 [select]: net/http.(*persistConn).writeLoop(0x400000c120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 44 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400028e770, 0x12c18bc, 0x1, 0x40001a1e58, 0x40001907e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x4000058ec8, 0x40001907e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40001a1fa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x40001e1700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 45 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x40001e1700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 7 [IO wait]: internal/poll.runtime_pollWait(0xffffbc21e110, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011e098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011e080, 0x4000162000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011e080, 0x4000162000, 0x1000, 0x1000, 0x4000069a88, 0x68f7a4, 0x40001ee1b0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010010, 0x4000162000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c120, 0x4000162000, 0x1000, 0x1000, 0xf12180, 0x400018e090, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400037ade0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400037ade0, 0xa, 0x1008880, 0x400018e050, 0x199, 0x1000, 0x4000069b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400037ade0, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400021c3f0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400021c3f0, 0x40000a2000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40001e0580, 0x40000a2000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40001e0580, 0x40000a2000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001e1580, 0x40000a2000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40001ee160, 0x0, 0x4000069eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40001ee160, 0x0, 0x0, 0x4000069f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40001ee160, 0xee0d40, 0x400018e050, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40001fe000, 0x40001c2a20, 0x4000190960) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 5 [IO wait]: internal/poll.runtime_pollWait(0xffffbc21e028, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000298118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000298100, 0x4000092000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000298100, 0x4000092000, 0x1000, 0x1000, 0x400009abe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000116018, 0x4000092000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400013a360, 0x4000092000, 0x1000, 0x1000, 0x400009acf8, 0x67867c, 0x4000190900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001c21e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40001c21e0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400013a360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 6 [select]: net/http.(*persistConn).writeLoop(0x400013a360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 48 [select]: net/http.(*persistConn).readLoop(0x400000c7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 49 [select]: net/http.(*persistConn).writeLoop(0x400000c7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 167 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40044690a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 10 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400011ad00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 169 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4004cd8400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 168 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40044690a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 68 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x4000574540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 80 [IO wait]: internal/poll.runtime_pollWait(0xffffbc21d718, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000210818, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000210800, 0x40044c5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000210800, 0x40044c5000, 0x1000, 0x1000, 0x400008b688, 0x69f178, 0x400008b6f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000116198, 0x40044c5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000306fc0, 0x40044c5000, 0x1000, 0x1000, 0x5, 0x400442c500, 0x400008b7c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001c39e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001c39e0, 0x68f90a, 0x400024a4d0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40001c39e0, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400024d0e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400024d0e0, 0x40024fe000, 0x1000, 0x1000, 0x400008b998, 0x44e884, 0x400008b948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40002523c0, 0x40024fe000, 0x1000, 0x1000, 0x4000256cb8, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40002523c0, 0x40024fe000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000252400, 0x40024fe000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400008bd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400008bd80, 0xa, 0x40024fe201, 0x1, 0xda7, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400008bd80, 0x40024fe20a, 0x1, 0x4000036826, 0x0, 0x40024fe259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400008bd80, 0x400027e90a, 0x4000036826, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000132d40, 0x40024cee40, 0x1f, 0x400008bee0, 0x40024cc438, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x4000132d40, 0x113075a, 0x7, 0x400050e6e0, 0xd, 0x40024cc438, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x4000132d00, 0x400050e6e0, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 74 [select]: net/http.(*persistConn).readLoop(0x400013bb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 71 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x4000574540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 72 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x4000574540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 73 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x4000574540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 75 [select]: net/http.(*persistConn).writeLoop(0x400013bb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 22 [IO wait]: internal/poll.runtime_pollWait(0xffffbc21d8e8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000299098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000299080, 0x400252c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000299080, 0x400252c000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000290408, 0x400252c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400013bb00, 0x400252c000, 0x1000, 0x1000, 0x400029d680, 0x200000003, 0x400029d680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400010f680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400010f680, 0x40000ac00a, 0x40001a78c8, 0x4d3b10, 0x40001c2060, 0x40000ac000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400010f680, 0x0, 0x1, 0x0, 0x40001c2078, 0x40000ac000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40004d7470) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40004d7470, 0x400457a000, 0x1000, 0x1000, 0x40001a79b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40002963c0, 0x400457a000, 0x1000, 0x1000, 0x40001c3318, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40002963c0, 0x400457a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000296400, 0x400457a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40001a7dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001a7dd0, 0xa, 0x400457a201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40001a7dd0, 0x400457a20a, 0x1, 0x400050e6c9, 0x0, 0x400457a2e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40001a7dd0, 0x4000204d0a, 0x400050e6c9, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000132d40, 0x40002c2480, 0x17, 0x40001a7f20, 0x400444f980, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x4000132d40, 0x113075a, 0x7, 0x400444f980, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x4000132d00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 23 [IO wait]: internal/poll.runtime_pollWait(0xffffbc21dab8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011ed18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011ed00, 0x4000529000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011ed00, 0x4000529000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000290288, 0x4000529000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c7e0, 0x4000529000, 0x1000, 0x1000, 0x400029d800, 0x200000003, 0x400029d800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400010e8a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400010e8a0, 0x40000ac00a, 0x40001a58c8, 0x4d3b10, 0x40001c2060, 0x40000ac000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400010e8a0, 0x0, 0x1, 0x0, 0x40001c2078, 0x40000ac000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40004d7050) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40004d7050, 0x40000a3000, 0x1000, 0x1000, 0x40001a59b8, 0x0, 0x40001a5988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000296240, 0x40000a3000, 0x1000, 0x1000, 0x4000182a80, 0x3, 0x40001a5be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000296240, 0x40000a3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000296280, 0x40000a3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40001a5dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001a5dd0, 0xa, 0x40000a3201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40001a5dd0, 0x40000a320a, 0x1, 0x400444c87b, 0x0, 0x40000a3263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40001a5dd0, 0x400011820a, 0x400444c87b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000132d40, 0x40000421e0, 0x24, 0x40001a5f20, 0x400444f968, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x4000132d40, 0x113075a, 0x7, 0x400444f968, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x4000132d00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 11 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x4000256540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 12 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x400024c360) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 13 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x40024ec000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 166 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40044690a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 131 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x40001c5900) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 130 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40024d2bc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 116 [IO wait]: internal/poll.runtime_pollWait(0xffffbc21d800, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40024ec498, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40024ec480, 0x4002488000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40024ec480, 0x4002488000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001c0558, 0x4002488000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400013aea0, 0x4002488000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000257740) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000257740, 0x100a, 0x0, 0x0, 0x0, 0x0, 0x40024f5438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000257740, 0x0, 0x40024f5478, 0x655784, 0x40024f54a8, 0x40b2f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40047beff0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40047beff0, 0x40047c4e00, 0x200, 0x200, 0x4, 0x0, 0x40024f5508) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40024d2f80, 0x40047c4e00, 0x200, 0x200, 0x4000113387, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40024d2f80, 0x40047c4e00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40024d2fc0, 0x40047c4e00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40047cda20, 0x680d00, 0x40024f56b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40047cda20, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40047cda20, 0xee0d40, 0x400018eaf0, 0x12db068, 0x40024d2fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40047ce870, 0x40047c8a80, 0x40047e2660) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 95 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400011b800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 61 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40024d2580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 83 [select]: net/http.(*persistConn).readLoop(0x4000306fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 84 [select]: net/http.(*persistConn).writeLoop(0x4000306fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 14 [IO wait]: internal/poll.runtime_pollWait(0xffffbc21d630, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40024ec118, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40024ec100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40024ec100, 0x0, 0x400015fe98, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40045685e8, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40045685e8, 0x12ed9d0, 0x40045685e8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x40024ec080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 15 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40045185b0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 16 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40045185b0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 98 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x40045185b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 99 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x400024ce40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 114 [IO wait]: internal/poll.runtime_pollWait(0xffffbc21d548, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000210e18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000210e00, 0x40047d0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000210e00, 0x40047d0000, 0x1000, 0x1000, 0x40044b5be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001163b0, 0x40047d0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000307d40, 0x40047d0000, 0x1000, 0x1000, 0x40044b5cf8, 0x67867c, 0x40002cb5c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40047c8540) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40047c8540, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4000307d40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 102 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40002b4070) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 65 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x4000297ac0, 0x4000167030, 0x40001670a0, 0x4004bdac40, 0x12e8f90, 0x40002082a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x4004bd5544, 0x5, 0x40001e1c00, 0x12eca70, 0x40002978c0, 0x130d478, 0x4000200380, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x400451a460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x400451a460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 64 [IO wait]: internal/poll.runtime_pollWait(0xffffbc21d460, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004754598, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4004754580, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4004754580, 0x12d9d48, 0x400011b880, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c88d80, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c88d80, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x40001e1c00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 105 [select]: net/http.(*persistConn).readLoop(0x400013aea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 106 [select]: net/http.(*persistConn).writeLoop(0x400013aea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 115 [select]: net/http.(*persistConn).writeLoop(0x4000307d40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 132 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x40001c5900) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 133 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x4000297ac0, 0x4004bdac40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 134 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40002082a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 135 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40002082a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 136 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x40024d4bd0, 0x400460f440) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 137 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x40024d2540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 138 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x40001c59a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 139 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x40001c59a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 140 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x40000103a8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 141 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x40001c59a0, 0x400460f4a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 142 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400028e770, 0x1144559, 0x1b, 0x40024fbe38, 0x400460f5c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004bdb760, 0x400460f5c0, 0xf1f700, 0x400028fff0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40024fbf80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4000483e60, 0x4004bdb760) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 143 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 144 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x40002aae60, 0xd00007, 0x4004c56578, 0x5, 0x4004bdb770, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 145 [IO wait]: internal/poll.runtime_pollWait(0xffffbc21d9d0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011f018, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400011f000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400011f000, 0x0, 0x40000a4d88, 0x665fe4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c01518, 0x611a6ce0, 0x40000a4db8, 0x445c24) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c01518, 0x40000a4e20, 0x40000a4e28, 0x18, 0x4004609800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x400451a540, 0x12ed9d0, 0x4004c01518, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x400451a540, 0x12ed9d0, 0x4004c01518, 0x4000202700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 146 [IO wait]: internal/poll.runtime_pollWait(0xffffbc21d378, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011f118, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400011f100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400011f100, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c01530, 0xffffbc30cf18, 0xffff941aa120, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c01530, 0x665e1c, 0x0, 0xfe5e60, 0x40024ef5c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004c01548, 0x4004c05a10, 0x11ae500, 0x400451a6a0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x400451a620, 0x12ec710, 0x4004c01548, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x400451a620, 0x12ec710, 0x4004c01548, 0x4000202700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 161 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x40045d5080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 165 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40044690a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 153 [IO wait]: internal/poll.runtime_pollWait(0xffffbc2630e8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011f498, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011f480, 0x4004d9b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011f480, 0x4004d9b000, 0x1000, 0x1000, 0x40000a6be0, 0x469b10, 0x4004609e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010458, 0x4004d9b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004c54900, 0x4004d9b000, 0x1000, 0x1000, 0x40000a6cf8, 0x67867c, 0x4004dbe000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004c6cc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004c6cc60, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004c54900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 164 [semacquire]: sync.runtime_Semacquire(0x4004df0778) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004df0770) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x40002e8680, 0x4000110570, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffc9442969, 0x29, 0xffffc94429cb, 0x25, 0x4004dbede0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x40047773d0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffc9442969, 0x29, 0xffffc94429cb, 0x25, 0x4004dbede0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x40047773d0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004d8b190, 0x4004d8ae60, 0x4000110570, 0x40001d3680, 0x40001e1c80, 0x0, 0x3f, 0x40044beda8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x4000202700, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x40047773d0, 0xb, 0x40047773c8, 0x6, 0x40047773e0, 0xd, 0x4004c8ac80, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x4000202700, 0x4000370000, 0x400019dc20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x4000202700, 0x40002582b0, 0x2, 0x2, 0x40002582c0, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 154 [select]: net/http.(*persistConn).writeLoop(0x4004c54900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 118 [IO wait]: internal/poll.runtime_pollWait(0xffffbc263000, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004754b98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004754b80, 0x4004ca7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004754b80, 0x4004ca7000, 0x1000, 0x1000, 0x4004dc3ad8, 0x412bc0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40002906c0, 0x4004ca7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001cfe60, 0x4004ca7000, 0x1000, 0x1000, 0x40047c8c00, 0x60, 0x58) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004e004e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004e004e0, 0x100a, 0x0, 0x0, 0x0, 0x0, 0x40024f5438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004e004e0, 0x0, 0x40024f5478, 0x655784, 0x40024f54a8, 0x40b2f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40047bf740) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40047bf740, 0x40047c5e00, 0x200, 0x200, 0x4, 0x0, 0x40024f5508) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40024d3380, 0x40047c5e00, 0x200, 0x200, 0x4000113387, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40024d3380, 0x40047c5e00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40024d33c0, 0x40047c5e00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40047f2c60, 0x680d00, 0x40024f56b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40047f2c60, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40047f2c60, 0xee0d40, 0x400018ec80, 0x12db068, 0x40024d33c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40047cef30, 0x40047c8f60, 0x40047e2f00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 170 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40045d51e0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 171 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40045d51e0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 172 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x40045d51e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 173 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40044690a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 174 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400028e770, 0x1144559, 0x1b, 0x400008de38, 0x4004cd6a20, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4000115ca0, 0x4004cd6a20, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400008df80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40044abe00, 0x4000115ca0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 179 [IO wait]: internal/poll.runtime_pollWait(0xffffbc262f18, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004754c98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004754c80, 0x4004e1b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004754c80, 0x4004e1b000, 0x1000, 0x1000, 0x1ac8f5ddebc, 0x0, 0x4004e09a98) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40002906e0, 0x4004e1b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*connReader).Read(0x4004ca5380, 0x4004e1b000, 0x1000, 0x1000, 0x705048333, 0x1be4a80, 0x72) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:800 +0x23c bufio.(*Reader).fill(0x4004e008a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004e008a0, 0x4, 0x705048333, 0x1be4a80, 0x0, 0x0, 0x1be4a80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*conn).serve(0x40005377c0, 0x12f86a0, 0x4004cd86c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:1977 +0x840 created by net/http.(*Server).Serve /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:3013 +0x318 goroutine 177 [select]: net/http.(*persistConn).readLoop(0x40001cfe60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 178 [select]: net/http.(*persistConn).writeLoop(0x40001cfe60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 160 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4004cd8440) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 183 [IO wait]: internal/poll.runtime_pollWait(0xffffbc262e30, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004754e18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004754e00, 0x4004e45000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004754e00, 0x4004e45000, 0x1000, 0x1000, 0x4004e14be0, 0x469b10, 0x4004cf8780) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000290700, 0x4004e45000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004e40240, 0x4004e45000, 0x1000, 0x1000, 0x4004e14cf8, 0x67867c, 0x4004cd7bc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004e00c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004e00c00, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004e40240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 184 [select]: net/http.(*persistConn).writeLoop(0x4004e40240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 194 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x40045d5080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 195 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x40045d5080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 196 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x40045d5080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 197 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004c07400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 198 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x4004c07400, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 199 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x4000202700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 200 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x4000202700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 201 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x4000202700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 202 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:49:25.705+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:49:25.705+00:00 [Info] Setting ipv6=false 2021-08-16T13:49:25.711+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:49:25.718+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:49:25.718+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:49:25.718+00:00 [Info] Setting log level to Info 2021-08-16T13:49:25.719+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:49:25.727+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:49:25 audit: created new audit service 2021-08-16T13:49:25.736+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:49:25.736+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:49:25.736+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:49:25.736+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:49:25.736+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:49:25.747+00:00 [Info] security context initialized 2021-08-16T13:49:25.747+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:49:25.747+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:49:25.747+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:49:25.748+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:49:25.748+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:49:25.760+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:49:25.764+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:49:25.766+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:49:25.786+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:49:25.787+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:49:25.788+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:49:25.790+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:49:25.791+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:49:25.795+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:49:25.795+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:49:25.796+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:49:25.839+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:49:25.840+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 53 us 2021-08-16T13:49:25.841+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:49:25.843+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:49:25.843+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:49:25.847+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:49:25.847+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:49:25.847+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:49:25.847+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:49:25.847+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:49:25.847+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:25.847+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:25.849+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:49:25.849+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:49:25.849+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:49:25.849+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:25.849+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:49:25.849+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:49:25.849+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:49:25.849+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:49:25.850+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:49:25.850+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:49:25.850+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:49:25.855+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 5.572291ms 2021-08-16T13:49:25.855+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:49:25.856+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:49:25.856+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:25.856+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:49:25.856+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:49:25.856+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:49:25.857+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:49:25.867+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:49:25.867+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:49:25.867+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:49:25.867+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:49:25.867+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x4004f0c25f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x4004f0c25f pc=0xbd8ea4] goroutine 188 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004f1e9f0 sp=0x4004f1e9c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004f1ea30 sp=0x4004f1e9f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x4004f2e000, 0x4004e0ac00, 0x11a9890, 0x4004f1eb40, 0x4004f2e018, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004f1eab0 sp=0x4004f1ea40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x4004f2e000, 0x4004e0ac00, 0x11a9890, 0x4004f1eb40, 0x4004f2e018, 0xc18f8c, 0x4004f1ec18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004f1eaf0 sp=0x4004f1eab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4000134c80, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004f1eda0 sp=0x4004f1eaf0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4000134c80, 0x4004423420, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004f1ee60 sp=0x4004f1eda0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x40047612b0, 0x40047612c0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004f1f200 sp=0x4004f1ee60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x40047612b0, 0x40047612c0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004f1f8f0 sp=0x4004f1f200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004f01080, 0x4004f0a380, 0x4004e4c580, 0x4004761290) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004f1ffb0 sp=0x4004f1f8f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004f1ffb0 sp=0x4004f1ffb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x400018a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x400020b560, 0x4, 0x1136a87, 0xf, 0x4000289e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 50 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400022e0d8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 51 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400022e0f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 52 [IO wait]: internal/poll.runtime_pollWait(0xffff64106898, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000198198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000198180, 0x40001a1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000198180, 0x40001a1000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400013e4f8, 0x40001a1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x4000140480, 0x40001a6001, 0xdff, 0xdff, 0x40001258e8, 0x6902d8, 0x4000285948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x40001305b8, 0x40001a6001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40001769a0, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40001769a0, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40001769a0, 0xf5a960, 0x4000140560, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4000140540, 0x4000150340, 0x4000285b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x4000152140, 0x12f8f98, 0x4000140540, 0x400013aca7, 0x16, 0x4000157560, 0x4000125c01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x4000152140, 0x12f8f98, 0x4000140540, 0x4000152140, 0x400013a4b0, 0x400013a4c0, 0x4000198200, 0x4000150260, 0xee0980, 0x4000157560, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x4000152140, 0x12f8f98, 0x4000140540) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x400020a4e0, 0x4000125f78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400011ff78, 0x400020a4e0, 0x12db8a8, 0x4000130588, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x400020a4e0, 0x4000224460, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x400020a4e0, 0x4000224460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 37 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400010c400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 53 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 54 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 55 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 56 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 57 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 58 [syscall]: syscall.Syscall(0x3f, 0x0, 0x4000145fa0, 0x4, 0x4000145e68, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x4000145fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000140000, 0x4000145fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x400013e000, 0x4000145fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 65 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400020e710, 0x12c18bc, 0x1, 0x4000125e58, 0x400010e0c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x40000556c8, 0x400010e0c0, 0x40000556c8, 0x679704) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000125fa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4000168100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 63 [select]: net/http.(*persistConn).readLoop(0x400000c240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 64 [select]: net/http.(*persistConn).writeLoop(0x400000c240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 66 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4000168100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 40 [IO wait]: internal/poll.runtime_pollWait(0xffff64105db8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001bc118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40001bc100, 0x40001be000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40001bc100, 0x40001be000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400013e310, 0x40001be000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c7e0, 0x40001be000, 0x1000, 0x1000, 0x400008af00, 0x200000003, 0x400008af00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000140de0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000140de0, 0x40014e400a, 0x40024f98f8, 0x412bc0, 0x4000140060, 0x40014e4000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000140de0, 0x0, 0x1, 0x3f, 0x4000140078, 0x40014e4000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4002527440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4002527440, 0x40002d9000, 0x1000, 0x1000, 0x40024f99b8, 0x44d8a8, 0x40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4002528940, 0x40002d9000, 0x1000, 0x1000, 0x400009abf8, 0x4002528a00, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4002528940, 0x40002d9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4002528980, 0x40002d9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40024f9dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40024f9dd0, 0xa, 0x40002d9201, 0x1, 0xd1f, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40024f9dd0, 0x40002d920a, 0x1, 0x4000036de9, 0x0, 0x40002d92e1, 0x1, 0xd1f, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40024f9dd0, 0x400252ca0a, 0x4000036de9, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400021aa00, 0x400003a240, 0x17, 0x40024f9f20, 0x40001ac228, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x400021aa00, 0x113075a, 0x7, 0x40001ac228, 0x0, 0x3461353062663237, 0x6138643865613537) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x400021a9c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 3 [IO wait]: internal/poll.runtime_pollWait(0xffff641066c8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000198418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000198400, 0x4000292000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000198400, 0x4000292000, 0x1000, 0x1000, 0x400029cbe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000216008, 0x4000292000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c6c0, 0x4000292000, 0x1000, 0x1000, 0x400029ccf8, 0x67867c, 0x400028e120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001401e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40001401e0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000c6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 4 [select]: net/http.(*persistConn).writeLoop(0x400000c6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 69 [IO wait]: internal/poll.runtime_pollWait(0xffff641067b0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000198318, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000198300, 0x40001a7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000198300, 0x40001a7000, 0x1000, 0x1000, 0x4000066a88, 0x68f7a4, 0x40002205d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400013e0d0, 0x40001a7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c240, 0x40001a7000, 0x1000, 0x1000, 0xf12180, 0x400009c1d0, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400009a360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400009a360, 0xa, 0x1008880, 0x400009c190, 0x199, 0xad9, 0x4000066b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400009a360, 0x2, 0x0, 0x9, 0x56b, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000280720) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000280720, 0x40014de000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000168580, 0x40014de000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000168580, 0x40014de000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000169580, 0x40014de000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4000220580, 0x0, 0x4000066eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4000220580, 0x0, 0x0, 0x4000066f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4000220580, 0xee0d40, 0x400009c190, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400023e000, 0x400007e2a0, 0x400023c900) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 24 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000169d40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 41 [IO wait]: internal/poll.runtime_pollWait(0xffff64105f88, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000198d98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000198d80, 0x4002537000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000198d80, 0x4002537000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000216290, 0x4002537000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40002757a0, 0x4002537000, 0x1000, 0x1000, 0x4000001680, 0x200000003, 0x4000001680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400007fce0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400007fce0, 0x40014e400a, 0x40024f78c8, 0x4d3b10, 0x4000140060, 0x40014e4000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400007fce0, 0x0, 0x1, 0x0, 0x4000140078, 0x40014e4000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4002527260) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4002527260, 0x40002d8000, 0x1000, 0x1000, 0x40024f79b8, 0x0, 0x40024f7988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4002528880, 0x40002d8000, 0x1000, 0x1000, 0x4000001500, 0x3, 0x40024f7be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4002528880, 0x40002d8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40025288c0, 0x40002d8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40024f7dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40024f7dd0, 0xa, 0x40002d8201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40024f7dd0, 0x40002d820a, 0x1, 0x4000036b9b, 0x0, 0x40002d8263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40024f7dd0, 0x400015040a, 0x4000036b9b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400021aa00, 0x4000042210, 0x24, 0x40024f7f20, 0x40001ac210, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x400021aa00, 0x113075a, 0x7, 0x40001ac210, 0x0, 0x2c305b2c5d312d2c, 0x2d2c305b2c5d312d) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x400021a9c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 9 [IO wait]: internal/poll.runtime_pollWait(0xffff64105ea0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000198598, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000198580, 0x40014df000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000198580, 0x40014df000, 0x1000, 0x1000, 0x40000e1688, 0x69f178, 0x40000e16f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010110, 0x40014df000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400441c6c0, 0x40014df000, 0x1000, 0x1000, 0x5, 0x40044aa500, 0x40000e17c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000483680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000483680, 0x68f90a, 0x400016ca50, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000483680, 0x0, 0x2, 0x0, 0x40000e1878, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40004c5410) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40004c5410, 0x4002534000, 0x1000, 0x1000, 0x40000e1998, 0x44e884, 0xffff00001001) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000214600, 0x4002534000, 0x1000, 0x1000, 0x40001405f8, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000214600, 0x4002534000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000214640, 0x4002534000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40000e1d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000e1d80, 0xa, 0x4002534201, 0x1, 0xffff8e9ca8f0, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40000e1d80, 0x400253420a, 0x1, 0x400465a0b0, 0x0, 0x4002534259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40000e1d80, 0x400452800a, 0x400465a0b0, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400021aa00, 0x40002a8520, 0x1f, 0x40000e1ee0, 0x40002aaa98, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x400021aa00, 0x113075a, 0x7, 0x4000036e00, 0xd, 0x40002aaa98, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x400021a9c0, 0x4000036e00, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 90 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40044286c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 74 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x40000e2540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 101 [select]: net/http.(*persistConn).writeLoop(0x400441c6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 100 [select]: net/http.(*persistConn).readLoop(0x400441c6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 13 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x40001a5040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 80 [select]: net/http.(*persistConn).readLoop(0x40002757a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 77 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x40000e2540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 78 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x40000e2540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 79 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x40000e2540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 81 [select]: net/http.(*persistConn).writeLoop(0x40002757a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 26 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40025281c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 27 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x400442a7e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 28 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x40001fc840) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 29 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x4000246f80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 30 [IO wait]: internal/poll.runtime_pollWait(0xffff64106070, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000247498, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000247480, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000247480, 0x0, 0x4000148698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x400022fa58, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x400022fa58, 0x12ed9d0, 0x400022fa58, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x4000247000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 31 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4004426a90, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 32 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4004426a90, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 33 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x4004426a90) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 82 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x40001fd320) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 83 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x400027e7e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 12 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x40001a5040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 117 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40002154c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 116 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4004428c80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 118 [IO wait]: internal/poll.runtime_pollWait(0xffff64105be8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001bc718, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40001bc700, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40001bc700, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c3b260, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c3b260, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x4000215c80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 44 [select]: net/http.(*persistConn).readLoop(0x400000c7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 45 [select]: net/http.(*persistConn).writeLoop(0x400000c7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 115 [select]: net/http.(*persistConn).writeLoop(0x400000ca20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 125 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40024c09a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 114 [select]: net/http.(*persistConn).readLoop(0x400000ca20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 124 [semacquire]: sync.runtime_Semacquire(0x4004f01088) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004f01080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x4004e4c580, 0x40004c4960, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xfffff032e969, 0x29, 0xfffff032e9cb, 0x25, 0x4004e06a20, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004c64be0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xfffff032e969, 0x29, 0xfffff032e9cb, 0x25, 0x4004e06a20, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004c64be0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004f1f190, 0x4004f1ee60, 0x40004c4960, 0x40000a46c0, 0x4000169d40, 0x0, 0xbf, 0x40014c4da8, 0xffff4e7d1b50, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x400018a000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004c64be0, 0xb, 0x4004c64bd8, 0x6, 0x4004c64bf0, 0xd, 0x4004c39280, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x400018a000, 0x4000480000, 0x40002801b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x400018a000, 0x40001fe7c0, 0x2, 0x2, 0x40001fe7d0, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 119 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x4002529c40, 0x4000242f50, 0x4000242fc0, 0x400006cd50, 0x12e8f90, 0x40024c0460, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x40046f1ef4, 0x5, 0x4000215c80, 0x12eca70, 0x4000215b40, 0x130d478, 0x40001881c0, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40024c08c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40024c08c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 14 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x4002529c40, 0x400006cd50) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 15 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40024c0460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 16 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40024c0460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 130 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x4004432d80, 0x4004651ce0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 131 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x4004428680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 132 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x40001a50e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 133 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x40001a50e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 134 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x40002164f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 135 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x40001a50e0, 0x4004651d40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 136 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400020e710, 0x1144559, 0x1b, 0x40024fbe38, 0x4004651e60, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x400006d960, 0x4004651e60, 0xf1f700, 0x400020ff90) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40024fbf80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004cea720, 0x400006d960) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 137 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 138 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400009cfa0, 0xd00007, 0x4004d2a318, 0x5, 0x400006d9a0, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 139 [IO wait]: internal/poll.runtime_pollWait(0xffff64106158, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40002af018, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40002af000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40002af000, 0xffff8e9c0108, 0x30, 0x4004d30120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40047fbc38, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40047fbc38, 0x4004d30120, 0x11ae500, 0x40002f0780, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40002f0700, 0x12ed9d0, 0x40047fbc38, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40002f0700, 0x12ed9d0, 0x40047fbc38, 0x400018a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 140 [IO wait]: internal/poll.runtime_pollWait(0xffff64105a18, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40002af118, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40002af100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40002af100, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40047fbc50, 0xffff8e9c0108, 0xffff65f65c40, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40047fbc50, 0x665e1c, 0x0, 0xfe5e60, 0x40014b95c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x40047fbc68, 0x4004d301b0, 0x11ae500, 0x40002f0860, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40002f07e0, 0x12ec710, 0x40047fbc68, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40002f07e0, 0x12ec710, 0x40047fbc68, 0x400018a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 126 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40024c09a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 123 [IO wait]: internal/poll.runtime_pollWait(0xffff64105b00, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001bc218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40001bc200, 0x4002482000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40001bc200, 0x4002482000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400013e450, 0x4002482000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000ca20, 0x4002482000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000141620) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000141620, 0xa, 0x0, 0x0, 0x0, 0x0, 0x40004f7438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000141620, 0x0, 0x40004f7478, 0x655784, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004c42e10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004c42e10, 0x4004c4a200, 0x200, 0x200, 0x4, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000215e00, 0x4004c4a200, 0x200, 0x200, 0x4000264187, 0x33, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000215e00, 0x4004c4a200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000215e40, 0x4004c4a200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004bef4a0, 0x0, 0x40004f76b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004bef4a0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004bef4a0, 0xee0d40, 0x4000152a50, 0x12db068, 0x4000215e40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40046c37a0, 0x40045671a0, 0x40045698c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 121 [IO wait]: internal/poll.runtime_pollWait(0xffff640653e8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001bc818, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40001bc800, 0x4004c4f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40001bc800, 0x4004c4f000, 0x1000, 0x1000, 0x400029bbe0, 0x469b10, 0x40002ddc80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400013e868, 0x4004c4f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004d2c120, 0x4004c4f000, 0x1000, 0x1000, 0x400029bcf8, 0x67867c, 0x4004569860) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004567020) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004567020, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004d2c120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 148 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40024c09a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 122 [select]: net/http.(*persistConn).writeLoop(0x4004d2c120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 94 [IO wait]: internal/poll.runtime_pollWait(0xffff64065300, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400475c418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400475c400, 0x4004753000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400475c400, 0x4004753000, 0x1000, 0x1000, 0x400254cbe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4004438298, 0x4004753000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001566c0, 0x4004753000, 0x1000, 0x1000, 0x400254ccf8, 0x67867c, 0x4004709ec0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004742cc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004742cc0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40001566c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 95 [select]: net/http.(*persistConn).writeLoop(0x40001566c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 127 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40024c09a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 128 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40024c09a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 149 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400020e710, 0x1144559, 0x1b, 0x4004d65e38, 0x4004569c20, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004d2eb10, 0x4004569c20, 0x12f86a0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004d65f80, 0x12edba8, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40045672c0, 0x4004d2eb10) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 129 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4000215fc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 162 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004e8e0b0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 163 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004e8e0b0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 164 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4004e8e0b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 152 [select]: net/http.(*persistConn).readLoop(0x4004d2c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 153 [select]: net/http.(*persistConn).writeLoop(0x4004d2c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 165 [IO wait]: internal/poll.runtime_pollWait(0xffff64105cd0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40002af618, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40002af600, 0x4004d81000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40002af600, 0x4004d81000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000216608, 0x4004d81000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004d2c5a0, 0x4004d81000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004cebc20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004cebc20, 0xa, 0x0, 0x0, 0x0, 0x0, 0x40004f7438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004cebc20, 0x0, 0x40004f7478, 0x655784, 0x40004f74c8, 0x412bc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004c43860) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004c43860, 0x4004c4a800, 0x200, 0x200, 0x4, 0x0, 0x40004f7578) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004e8a140, 0x4004c4a800, 0x200, 0x200, 0x4000264100, 0x33, 0xffff65f3e538) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004e8a140, 0x4004c4a800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004e8a180, 0x4004c4a800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004e982c0, 0x4000480000, 0x40004f76b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004e982c0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004e982c0, 0xee0d40, 0x4000152cd0, 0x12db068, 0x4004e8a180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004e9a000, 0x40045678c0, 0x4004e8c6c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 97 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4004e8a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 178 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x400016def0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 179 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x400016def0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 180 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x400016def0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 181 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x400016def0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 182 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004741c00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 183 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x4004741c00, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 184 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x400018a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 185 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x400018a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 186 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x400018a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 187 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:49:31.079+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:49:31.080+00:00 [Info] Setting ipv6=false 2021-08-16T13:49:31.095+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:49:31.096+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:49:31.096+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:49:31.096+00:00 [Info] Setting log level to Info 2021-08-16T13:49:31.096+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:49:31.099+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:49:31 audit: created new audit service 2021-08-16T13:49:31.107+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:49:31.107+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:49:31.108+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:49:31.108+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:49:31.108+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:49:31.131+00:00 [Info] security context initialized 2021-08-16T13:49:31.131+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:49:31.131+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:49:31.131+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:49:31.131+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:49:31.132+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:49:31.145+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:49:31.145+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:49:31.156+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:49:31.173+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:49:31.173+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:49:31.175+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:49:31.176+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:49:31.177+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:49:31.178+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:49:31.178+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:49:31.179+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:49:31.229+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:49:31.230+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 50 us 2021-08-16T13:49:31.230+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:49:31.232+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:49:31.232+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:49:31.236+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:49:31.236+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:49:31.236+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:49:31.236+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:49:31.236+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:49:31.236+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:31.236+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:31.237+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:49:31.237+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:49:31.237+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:49:31.237+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:31.238+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:49:31.238+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:49:31.238+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:49:31.238+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:49:31.238+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:49:31.238+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:49:31.238+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:49:31.243+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 5.323579ms 2021-08-16T13:49:31.243+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:49:31.246+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:49:31.246+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:31.246+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:49:31.246+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:49:31.246+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:49:31.246+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... unexpected fault address 0x4004d8c49f fatal error: fault 2021-08-16T13:49:31.258+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:49:31.258+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:49:31.258+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:49:31.259+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:49:31.259+00:00 [Info] : Recover shard shard1 from metadata completed. [signal SIGBUS: bus error code=0x1 addr=0x4004d8c49f pc=0xbd8ea4] goroutine 181 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004bc49f0 sp=0x4004bc49c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004bc4a30 sp=0x4004bc49f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x4004dc61a0, 0x4002571800, 0x11a9890, 0x4004bc4b40, 0x4004dc61b8, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004bc4ab0 sp=0x4004bc4a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x4004dc61a0, 0x4002571800, 0x11a9890, 0x4004bc4b40, 0x4004dc61b8, 0xc18f8c, 0x4004bc4c18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004bc4af0 sp=0x4004bc4ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x400256c500, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004bc4da0 sp=0x4004bc4af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x400256c500, 0x4000215960, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004bc4e60 sp=0x4004bc4da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004dce0c0, 0x4004dce0d0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004bc5200 sp=0x4004bc4e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004dce0c0, 0x4004dce0d0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004bc58f0 sp=0x4004bc5200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004dd6800, 0x4004675c00, 0x4004da6580, 0x4004dce0a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004bc5fb0 sp=0x4004bc58f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004bc5fb0 sp=0x4004bc5fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x40000c6380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x400020d290, 0x4, 0x1136a87, 0xf, 0x4000095e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 20 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x40001aa660) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 21 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x40001aa678) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 22 [IO wait]: internal/poll.runtime_pollWait(0xffff58a4ae88, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000096098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000096080, 0x40000a7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000096080, 0x40000a7000, 0x1000, 0x1000, 0x1311860, 0x47b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010020, 0x40000a7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x4000082240, 0x40000b6001, 0xdff, 0xdff, 0x400019d8e8, 0x6902d8, 0x4000091948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400000e060, 0x40000b6001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40000a4160, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40000a4160, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40000a4160, 0xf5a960, 0x4000082320, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4000082300, 0x4000046160, 0x4000091b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x40000b0000, 0x12f8f98, 0x4000082300, 0x40000368e7, 0x16, 0x400000c5a0, 0x400019dc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x40000b0000, 0x12f8f98, 0x4000082300, 0x40000b0000, 0x40000360f0, 0x4000036100, 0x4000096100, 0x4000046080, 0xee0980, 0x400000c5a0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x40000b0000, 0x12f8f98, 0x4000082300) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x400020c210, 0x400019df78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400019ff78, 0x400020c210, 0x12db8a8, 0x400000e030, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x400020c210, 0x40001bd220, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x400020c210, 0x40001bd220) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 23 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 24 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 25 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 26 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 27 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 34 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400010c100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 28 [syscall]: syscall.Syscall(0x3f, 0x0, 0x4000404fa0, 0x4, 0x4000404e68, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x4000404fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x40001ba000, 0x4000404fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x40001b8000, 0x4000404fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 32 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400022c360, 0x12c18bc, 0x1, 0x4000475e58, 0x4000122060, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x4000400ec8, 0x4000122060, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000475fa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x400038e200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 52 [select]: net/http.(*persistConn).readLoop(0x40003be900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 53 [select]: net/http.(*persistConn).writeLoop(0x40003be900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 33 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x400038e200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 78 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x40024ac380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 68 [IO wait]: internal/poll.runtime_pollWait(0xffff58a4acb8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000096218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000096200, 0x400047e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000096200, 0x400047e000, 0x1000, 0x1000, 0x400047bbe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000100e0, 0x400047e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c5a0, 0x400047e000, 0x1000, 0x1000, 0x400047bcf8, 0x67867c, 0x40000384e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000118900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000118900, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 69 [select]: net/http.(*persistConn).writeLoop(0x400000c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 35 [IO wait]: internal/poll.runtime_pollWait(0xffff58a4ada0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000208198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000208180, 0x40000b7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000208180, 0x40000b7000, 0x1000, 0x1000, 0x4000477a88, 0x68f7a4, 0x400013e050) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b80e0, 0x40000b7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003be900, 0x40000b7000, 0x1000, 0x1000, 0xf12180, 0x40000b00e0, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003bd320) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003bd320, 0xa, 0x1008880, 0x40000b00a0, 0x199, 0x1000, 0x4000477b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003bd320, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000194d50) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000194d50, 0x4000144000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000080000, 0x4000144000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000080000, 0x4000144000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000080040, 0x4000144000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x400013e000, 0x0, 0x4000477eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400013e000, 0x0, 0x0, 0x4000477f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400013e000, 0xee0d40, 0x40000b00a0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4000124000, 0x4000082420, 0x4000134000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 16 [select]: net/http.(*persistConn).readLoop(0x40000f85a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 99 [IO wait]: internal/poll.runtime_pollWait(0xffff58a4a2c0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40024ff618, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40024ff600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40024ff600, 0x58ec48, 0x40044136e0, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004670e10, 0x4714f4, 0x40044136e0, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004670e10, 0x400246b500, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x4000117ec0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 7 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000116f00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 81 [IO wait]: internal/poll.runtime_pollWait(0xffff58a4a1d8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000096398, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000096380, 0x4004441000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000096380, 0x4004441000, 0x1000, 0x1000, 0x4000439688, 0x69f178, 0x40004396f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000ee060, 0x4004441000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40024527e0, 0x4004441000, 0x1000, 0x1000, 0x5, 0x400255e000, 0x40004397c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40000831a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000831a0, 0x68f90a, 0x40000bca50, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40000831a0, 0x0, 0x4000439868, 0x41a0a8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004422fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004422fc0, 0x4002553000, 0x1000, 0x1000, 0x4000439998, 0x44e884, 0x1c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40001168c0, 0x4002553000, 0x1000, 0x1000, 0x40024755b8, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40001168c0, 0x4002553000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000116900, 0x4002553000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000439d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000439d80, 0xa, 0x4002553201, 0x1, 0xffff83241110, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000439d80, 0x400255320a, 0x1, 0x40045a3250, 0x0, 0x4002553259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000439d80, 0x40000a560a, 0x40045a3250, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400042ec70, 0x400443ed20, 0x1f, 0x4000439ee0, 0x40000aa5a0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x400042ec70, 0x113075a, 0x7, 0x4002524ce0, 0xd, 0x40000aa5a0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x400042ec30, 0x4002524ce0, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 9 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x40024ac380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 40 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400252a100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 116 [select]: net/http.(*persistConn).readLoop(0x40024527e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 58 [IO wait]: internal/poll.runtime_pollWait(0xffff428d1428, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004e3e018, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004e3e000, 0x4004e40000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004e3e000, 0x4004e40000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400014e668, 0x4004e40000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004d1c360, 0x4004e40000, 0x1000, 0x1000, 0x20, 0x30, 0x20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004e3c7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004e3c7e0, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004e3c7e0, 0x41ab8c, 0xffff4298b700, 0x100100000000, 0xffff42b2cfff, 0x400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40046fd4d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40046fd4d0, 0x4004bb1000, 0x200, 0x200, 0x4004447d08, 0x41cff4, 0xffff4298b700) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400038fc40, 0x4004bb1000, 0x200, 0x200, 0xffff435b86b8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400038fc40, 0x4004bb1000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400038fc80, 0x4004bb1000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004bce000, 0x0, 0x4004447eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004bce000, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004bce000, 0xee0d40, 0x4000180c30, 0x12db068, 0x400038fc80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40046ff8c0, 0x40046b32c0, 0x4004add4a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 96 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000080b00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 79 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x40024ac380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 80 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x40024ac380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 41 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x4002512540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 42 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x40000badb0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 43 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x40000b8800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 44 [IO wait]: internal/poll.runtime_pollWait(0xffff58a4a660, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000b8918, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000b8900, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000b8900, 0x0, 0x4000058698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x400011b560, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x400011b560, 0x12ed9d0, 0x400011b560, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x40000b8880) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 45 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400042e8f0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 46 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400042e8f0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 47 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x400042e8f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 48 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x40000bb890) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 86 [IO wait]: internal/poll.runtime_pollWait(0xffff58a4a748, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40024fea18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40024fea00, 0x4002551000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40024fea00, 0x4002551000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000101d8, 0x4002551000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40000f85a0, 0x4002551000, 0x1000, 0x1000, 0x4004482d80, 0x200000003, 0x4004482d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40024753e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40024753e0, 0x400001400a, 0x400043d8c8, 0x4d3b10, 0x40001ba060, 0x4000014000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40024753e0, 0x0, 0x1, 0x0, 0x40001ba078, 0x4000014000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400010eae0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400010eae0, 0x400001a000, 0x1000, 0x1000, 0x400043d9b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000080cc0, 0x400001a000, 0x1000, 0x1000, 0x4000082958, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000080cc0, 0x400001a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000080d00, 0x400001a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400043ddd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400043ddd0, 0xa, 0x400001a201, 0x1, 0xd1f, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400043ddd0, 0x400001a20a, 0x1, 0x4002524cc9, 0x0, 0x400001a2e1, 0x1, 0xd1f, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400043ddd0, 0x400019760a, 0x4002524cc9, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400042ec70, 0x400012e870, 0x17, 0x400043df20, 0x400011b788, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x400042ec70, 0x113075a, 0x7, 0x400011b788, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x400042ec30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 84 [select]: net/http.(*persistConn).readLoop(0x40001c6360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 83 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40001f82a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 85 [select]: net/http.(*persistConn).writeLoop(0x40001c6360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 98 [select]: net/http.(*persistConn).writeLoop(0x40000f85a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 14 [select]: net/http.(*persistConn).readLoop(0x40000f86c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 15 [select]: net/http.(*persistConn).writeLoop(0x40000f86c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 87 [IO wait]: internal/poll.runtime_pollWait(0xffff58a4a3a8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40024feb98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40024feb80, 0x400254f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40024feb80, 0x400254f000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000101c8, 0x400254f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40000f86c0, 0x400254f000, 0x1000, 0x1000, 0x4004483380, 0x200000003, 0x4004483380) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002475320) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002475320, 0x400001400a, 0x400019b8c8, 0x4d3b10, 0x40001ba060, 0x4000014000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4002475320, 0x0, 0x1, 0x0, 0x40001ba078, 0x4000014000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400010eb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400010eb40, 0x400001b000, 0x1000, 0x1000, 0x400019b9b8, 0x0, 0x400019b988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000080d40, 0x400001b000, 0x1000, 0x1000, 0x4000187980, 0x3, 0x400019bbe0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000080d40, 0x400001b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000080d80, 0x400001b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400019bdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400019bdd0, 0xa, 0x400001b201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400019bdd0, 0x400001b20a, 0x1, 0x40044d3c9b, 0x0, 0x400001b263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400019bdd0, 0x400013250a, 0x40044d3c9b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400042ec70, 0x400012aa80, 0x24, 0x400019bf20, 0x400011b770, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x400042ec70, 0x113075a, 0x7, 0x400011b770, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x400042ec30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 104 [IO wait]: internal/poll.runtime_pollWait(0xffff428d1258, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000209918, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000209900, 0x4004657000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000209900, 0x4004657000, 0x1000, 0x1000, 0x400247abe0, 0x469b10, 0x4004483980) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000106b8, 0x4004657000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003bf9e0, 0x4004657000, 0x1000, 0x1000, 0x400247acf8, 0x67867c, 0x40046764e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40046348a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40046348a0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40003bf9e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 106 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4004d44540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 54 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40000811c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 117 [select]: net/http.(*persistConn).writeLoop(0x40024527e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 133 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400038f280) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 131 [IO wait]: internal/poll.runtime_pollWait(0xffff58a4a578, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000b9098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000b9080, 0x40045d2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000b9080, 0x40045d2000, 0x1000, 0x1000, 0x40045cfbe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400014e200, 0x40045d2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c7680, 0x40045d2000, 0x1000, 0x1000, 0x40045cfcf8, 0x67867c, 0x4000123200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001ba840) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40001ba840, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40001c7680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 132 [select]: net/http.(*persistConn).writeLoop(0x40001c7680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 100 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x4000081f00, 0x4000215110, 0x4000215180, 0x40045d9510, 0x12e8f90, 0x400457e7e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x40045fff24, 0x5, 0x4000117ec0, 0x12eca70, 0x4000081d00, 0x130d478, 0x40000c4000, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40024ac9a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40024ac9a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 134 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x400015d2c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 135 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x400015d2c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 136 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x4000081f00, 0x40045d9510) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 137 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x400457e7e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 138 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x400457e7e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 139 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x40001259e0, 0x4004b54d80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 140 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x4000080ac0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 141 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x400015d360) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 142 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x400015d360) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 143 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x400014e460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 144 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x400015d360, 0x4004b54de0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 145 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400022c360, 0x1144559, 0x1b, 0x4004ad9e38, 0x4004b54f00, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004cca030, 0x4004b54f00, 0xf1f700, 0x400022dbe0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004ad9f80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004cbe720, 0x4004cca030) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 146 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 147 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x40000f0e10, 0xd00007, 0x4004b7af58, 0x5, 0x4004cca040, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 148 [IO wait]: internal/poll.runtime_pollWait(0xffff58a4a0f0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000b9818, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000b9800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000b9800, 0xffff832cd5b8, 0x30, 0x4004b39fb0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004b74a38, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004b74a38, 0x4004b39fb0, 0x11ae500, 0x40024acb00, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40024aca80, 0x12ed9d0, 0x4004b74a38, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40024aca80, 0x12ed9d0, 0x4004b74a38, 0x40000c6380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 149 [IO wait]: internal/poll.runtime_pollWait(0xffff58a4a008, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000b9918, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000b9900, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000b9900, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004b74a50, 0xffff832cd5b8, 0xffff428fd538, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004b74a50, 0x665e1c, 0x0, 0xfe5e60, 0x40045cddc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004b74a68, 0x4004cd6060, 0x11ae500, 0x40024acbe0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40024acb60, 0x12ec710, 0x4004b74a68, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40024acb60, 0x12ec710, 0x4004b74a68, 0x40000c6380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 166 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40024ace00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 164 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40024ace00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 156 [IO wait]: internal/poll.runtime_pollWait(0xffff428d1340, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000b9c98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000b9c80, 0x4004cfe000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000b9c80, 0x4004cfe000, 0x1000, 0x1000, 0x40045cebe0, 0x469b10, 0x4004b7fb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400014e4f8, 0x4004cfe000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000dc20, 0x4004cfe000, 0x1000, 0x1000, 0x40045cecf8, 0x67867c, 0x4004b55680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004cbf200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004cbf200, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000dc20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 157 [select]: net/http.(*persistConn).writeLoop(0x400000dc20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 165 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40024ace00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 163 [semacquire]: sync.runtime_Semacquire(0x4004dd6808) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004dd6800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x4004da6580, 0x400008c4b0, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xfffffeb69969, 0x29, 0xfffffeb699cb, 0x25, 0x4004676e40, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x400467a690, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xfffffeb69969, 0x29, 0xfffffeb699cb, 0x25, 0x4004676e40, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x400467a690, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004bc5190, 0x4004bc4e60, 0x400008c4b0, 0x40003f7d40, 0x4000116f00, 0x0, 0x3f, 0x4004cdeda8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x40000c6380, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x400467a690, 0xb, 0x400467a688, 0x6, 0x400467a6a0, 0xd, 0x4004673280, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x40000c6380, 0x40003b0000, 0x40003fc150) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x40000c6380, 0x40000b2b00, 0x2, 0x2, 0x40000b2b10, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 103 [IO wait]: internal/poll.runtime_pollWait(0xffff58a4a490, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000b8a98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000b8a80, 0x40044f8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000b8a80, 0x40044f8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400014e130, 0x40044f8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c6360, 0x40044f8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002512ea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002512ea0, 0xa, 0x40003c0123, 0x11320c7, 0x9, 0x1, 0x4000136bf4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4002512ea0, 0xd4fa28, 0x40001b2200, 0x46, 0x11320c7, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400460d620) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400460d620, 0x4004615800, 0x200, 0x200, 0x400467a770, 0x400467a770, 0x40000e27e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004d80200, 0x4004615800, 0x200, 0x200, 0x4004413bc0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004d80200, 0x4004615800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004d80240, 0x4004615800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x400464f600, 0x680d00, 0x4000136eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400464f600, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400464f600, 0xee0d40, 0x400011cf50, 0x12db068, 0x4004d80240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400467c510, 0x40046347e0, 0x40046763c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 167 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40024ace00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 168 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4004d44500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 169 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004b76e70, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 170 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004b76e70, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 171 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4004b76e70) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 172 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40024ace00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 173 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400022c360, 0x1144559, 0x1b, 0x4004ad5e38, 0x4004d76780, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004ccad80, 0x4004d76780, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004ad5f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004cbfda0, 0x4004ccad80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 176 [select]: net/http.(*persistConn).readLoop(0x4004d1c360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 177 [select]: net/http.(*persistConn).writeLoop(0x4004d1c360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 105 [select]: net/http.(*persistConn).writeLoop(0x40003bf9e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 107 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x4004b76d10) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 108 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x4004b76d10) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 109 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x4004b76d10) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 110 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x4004b76d10) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 111 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004649200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 112 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x4004649200, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 113 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x40000c6380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 178 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x40000c6380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 179 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x40000c6380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 180 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:49:36.481+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:49:36.481+00:00 [Info] Setting ipv6=false 2021-08-16T13:49:36.499+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:49:36.500+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:49:36.500+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:49:36.500+00:00 [Info] Setting log level to Info 2021-08-16T13:49:36.500+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:49:36.503+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:49:36 audit: created new audit service 2021-08-16T13:49:36.512+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:49:36.512+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:49:36.513+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:49:36.513+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:49:36.513+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:49:36.526+00:00 [Info] security context initialized 2021-08-16T13:49:36.526+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:49:36.526+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:49:36.526+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:49:36.526+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:49:36.526+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:49:36.540+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:49:36.541+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:49:36.554+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:49:36.568+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:49:36.568+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:49:36.570+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:49:36.571+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:49:36.572+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:49:36.573+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:49:36.574+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:49:36.574+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:49:36.626+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:49:36.626+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 54 us 2021-08-16T13:49:36.626+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:49:36.629+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:49:36.629+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:49:36.636+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:49:36.636+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:49:36.636+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:49:36.636+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:49:36.636+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:49:36.636+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:36.636+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:36.637+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:49:36.637+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:49:36.637+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:49:36.637+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:36.638+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:49:36.638+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:49:36.638+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:49:36.638+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:49:36.638+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:49:36.638+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:49:36.638+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:49:36.645+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 6.894021ms 2021-08-16T13:49:36.645+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:49:36.646+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:49:36.646+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:36.646+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:49:36.646+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:49:36.646+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:49:36.646+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:49:36.657+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:49:36.657+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:49:36.657+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:49:36.657+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:49:36.657+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x4004e6001f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x4004e6001f pc=0xbd8ea4] goroutine 172 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x400236a9f0 sp=0x400236a9c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x400236aa30 sp=0x400236a9f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x4004704ea0, 0x4004e64000, 0x11a9890, 0x400236ab40, 0x4004704eb8, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x400236aab0 sp=0x400236aa40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x4004704ea0, 0x4004e64000, 0x11a9890, 0x400236ab40, 0x4004704eb8, 0xc18f8c, 0x400236ac18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x400236aaf0 sp=0x400236aab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4000366f00, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x400236ada0 sp=0x400236aaf0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4000366f00, 0x4002527f10, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x400236ae60 sp=0x400236ada0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004c1edc0, 0x4004c1edd0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x400236b200 sp=0x400236ae60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004c1edc0, 0x4004c1edd0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x400236b8f0 sp=0x400236b200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004e450e0, 0x4004e46e00, 0x40000d2100, 0x4004c1eda0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x400236bfb0 sp=0x400236b8f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x400236bfb0 sp=0x400236bfb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x400014e380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x40000fb6e0, 0x4, 0x1136a87, 0xf, 0x40000f9e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 7 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400000e690) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 8 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400000e6a8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 9 [IO wait]: internal/poll.runtime_pollWait(0xffff68871198, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0898, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0880, 0x40001a3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0880, 0x40001a3000, 0x1000, 0x1000, 0x1311860, 0x47b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000105b0, 0x40001a3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x4000082cc0, 0x4000116001, 0xdff, 0xdff, 0x400009f8e8, 0x6902d8, 0x40000f5948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400000e6f0, 0x4000116001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40000ce840, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40000ce840, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40000ce840, 0xf5a960, 0x4000082da0, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4000082d80, 0x40001280c0, 0x40000f5b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400008c820, 0x12f8f98, 0x4000082d80, 0x400010e7d7, 0x16, 0x400011c120, 0x400009fc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400008c820, 0x12f8f98, 0x4000082d80, 0x400008c820, 0x4000036c58, 0x4000036c60, 0x40000f0900, 0x40000466e0, 0xee0980, 0x400011c120, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400008c820, 0x12f8f98, 0x4000082d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x40000fa3c0, 0x400009ff78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400009df78, 0x40000fa3c0, 0x12db8a8, 0x400000e6c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x40000fa3c0, 0x40000b1360, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x40000fa3c0, 0x40000b1360) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 10 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 19 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400010c100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 11 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 12 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 13 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 14 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 15 [syscall]: syscall.Syscall(0x3f, 0x0, 0x400037bfa0, 0x4, 0x400037be68, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x400037bfa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x400037bfa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x400037bfa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 39 [select]: net/http.(*persistConn).writeLoop(0x400031e900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 36 [select]: net/http.(*persistConn).readLoop(0x400011c120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 37 [select]: net/http.(*persistConn).writeLoop(0x400011c120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 23 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001944c0, 0x12c18bc, 0x1, 0x400009fe58, 0x40023446c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x40003766c8, 0x40023446c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400009ffa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4000138040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 24 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4000138040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 44 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40023ba980) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 38 [IO wait]: internal/poll.runtime_pollWait(0xffff68870fc8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0080, 0x40003d2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0080, 0x40003d2000, 0x1000, 0x1000, 0x400006abe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000122078, 0x40003d2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400031e900, 0x40003d2000, 0x1000, 0x1000, 0x400006acf8, 0x67867c, 0x4002344c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000148a80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000148a80, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400031e900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 27 [IO wait]: internal/poll.runtime_pollWait(0xffff688710b0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400014c198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400014c180, 0x4000117000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400014c180, 0x4000117000, 0x1000, 0x1000, 0x4000160a88, 0x68f7a4, 0x400001e310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000122048, 0x4000117000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400011c120, 0x4000117000, 0x1000, 0x1000, 0xf12180, 0x400008c360, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001487e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001487e0, 0xa, 0x1008880, 0x400008c320, 0x199, 0xad9, 0x4000160b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40001487e0, 0x2, 0x0, 0x9, 0x56b, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000372ab0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000372ab0, 0x400012c000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40001f6440, 0x400012c000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40001f6440, 0x400012c000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001f6480, 0x400012c000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x400001e2c0, 0x0, 0x4000160eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400001e2c0, 0x0, 0x0, 0x4000160f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400001e2c0, 0xee0d40, 0x400008c320, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40000df680, 0x4000331680, 0x4000038600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 54 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001f6ac0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 72 [select]: net/http.(*persistConn).writeLoop(0x400011c900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 71 [select]: net/http.(*persistConn).readLoop(0x400011c900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 117 [IO wait]: internal/poll.runtime_pollWait(0xffff688704e8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4002397398, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4002397380, 0x40023f0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4002397380, 0x40023f0000, 0x1000, 0x1000, 0x40000f3688, 0x69f178, 0x40000f36f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001221c0, 0x40023f0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400248cd80, 0x40023f0000, 0x1000, 0x1000, 0x5, 0x4004408000, 0x40000f37c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400235be00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400235be00, 0x68f90a, 0x40013bec60, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400235be00, 0x0, 0x40000f3868, 0x41a0a8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400247b9e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400247b9e0, 0x40003d1000, 0x1000, 0x1000, 0x40000f3998, 0x44e884, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40001f6d80, 0x40003d1000, 0x1000, 0x1000, 0x40001499d8, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40001f6d80, 0x40003d1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001f6dc0, 0x40003d1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40000f3d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000f3d80, 0xa, 0x40003d1201, 0x1, 0xffff686c38f0, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40000f3d80, 0x40003d120a, 0x1, 0x40044c1dd0, 0x0, 0x40003d1259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40000f3d80, 0x40013fab0a, 0x40044c1dd0, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40013c05f0, 0x40023a2a20, 0x1f, 0x40000f3ee0, 0x40023e2558, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x40013c05f0, 0x113075a, 0x7, 0x40023c6060, 0xd, 0x40023e2558, 0x0, 0x0, 0x2) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x40013c05b0, 0x40023c6060, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 56 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x40000e87e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 66 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x40000e87e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 67 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x40000e87e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 68 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x40000e87e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 59 [select]: net/http.(*persistConn).readLoop(0x400248c7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 60 [select]: net/http.(*persistConn).writeLoop(0x400248c7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 31 [IO wait]: internal/poll.runtime_pollWait(0xffff68870888, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4002396998, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4002396980, 0x40023c1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4002396980, 0x40023c1000, 0x1000, 0x1000, 0x40003dabe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000122190, 0x40023c1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400248ca20, 0x40023c1000, 0x1000, 0x1000, 0x40003dacf8, 0x67867c, 0x40023c8000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400235b080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x400235b080, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400248ca20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 77 [IO wait]: internal/poll.runtime_pollWait(0xffff688706b8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0598, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0580, 0x40013ef000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0580, 0x40013ef000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40013b80d0, 0x40013ef000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400011cd80, 0x40013ef000, 0x1000, 0x1000, 0x4000103380, 0x200000003, 0x4000103380) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000149e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000149e00, 0x400030e00a, 0x400009b8c8, 0x4d3b10, 0x40000820c0, 0x400030e000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000149e00, 0x0, 0x1, 0x0, 0x40000820d8, 0x400030e000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40013ab0e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40013ab0e0, 0x40013fe000, 0x1000, 0x1000, 0x400009b9b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40013ba680, 0x40013fe000, 0x1000, 0x1000, 0x40001499d8, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40013ba680, 0x40013fe000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40013ba6c0, 0x40013fe000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400009bdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400009bdd0, 0xa, 0x40013fe201, 0x1, 0xd1f, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400009bdd0, 0x40013fe20a, 0x1, 0x4000037ff9, 0x0, 0x40013fe2e1, 0x1, 0xd1f, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400009bdd0, 0x400012770a, 0x4000037ff9, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40013c05f0, 0x40013a84b0, 0x17, 0x400009bf20, 0x40013cab28, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x40013c05f0, 0x113075a, 0x7, 0x40013cab28, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x40013c05b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 45 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x400247c900) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 46 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x400247aba0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 47 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x400014c100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 48 [IO wait]: internal/poll.runtime_pollWait(0xffff68870a58, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400014c418, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400014c400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400014c400, 0x0, 0x4000054698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40003cf038, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40003cf038, 0x12ed9d0, 0x40003cf038, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x400014c380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 49 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4002508ea0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 82 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4002508ea0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 83 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x4002508ea0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 84 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x400247b680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 28 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40000e01c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 78 [IO wait]: internal/poll.runtime_pollWait(0xffff68870970, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4002397218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4002397200, 0x40024e2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4002397200, 0x40024e2000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4002362418, 0x40024e2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400248c7e0, 0x40024e2000, 0x1000, 0x1000, 0x4000103500, 0x200000003, 0x4000103500) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400235b8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400235b8c0, 0x400030e00a, 0x40000a18c8, 0x4d3b10, 0x40000820c0, 0x400030e000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400235b8c0, 0x0, 0x1, 0x0, 0x40000820d8, 0x400030e000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40013ab080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40013ab080, 0x40013fd000, 0x1000, 0x1000, 0x40000a19b8, 0x0, 0x40000a1988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40013ba600, 0x40013fd000, 0x1000, 0x1000, 0x4000368000, 0x3, 0x40000a1be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40013ba600, 0x40013fd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40013ba640, 0x40013fd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40000a1dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000a1dd0, 0xa, 0x40013fd201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40000a1dd0, 0x40013fd20a, 0x1, 0x4000037e1b, 0x0, 0x40013fd263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40000a1dd0, 0x400012850a, 0x4000037e1b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40013c05f0, 0x40013a2900, 0x24, 0x40000a1f20, 0x40013cab10, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x40013c05f0, 0x113075a, 0x7, 0x40013cab10, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x40013c05b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 101 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x40013c6dc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 81 [select]: net/http.(*persistConn).readLoop(0x400011cd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 98 [select]: net/http.(*persistConn).writeLoop(0x400011cd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 32 [select]: net/http.(*persistConn).writeLoop(0x400248ca20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 100 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x40013c6dc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 99 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001f7c40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 122 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000138980) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 120 [select]: net/http.(*persistConn).readLoop(0x400248cd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 121 [select]: net/http.(*persistConn).writeLoop(0x400248cd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 85 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001390c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 123 [IO wait]: internal/poll.runtime_pollWait(0xffff688705d0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400477e398, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400477e380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400477e380, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c06288, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c06288, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x4004776500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 124 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x40013bb740, 0x4002527260, 0x40025272d0, 0x4004ba0820, 0x12e8f90, 0x4000114620, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x4004561284, 0x5, 0x4004776500, 0x12eca70, 0x40013bb500, 0x130d478, 0x40000e02a0, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40024847e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40024847e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 102 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x40013bb740, 0x4004ba0820) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 103 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x4000114620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 104 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x4000114620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 105 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x400011b680, 0x40045196e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 106 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x4000138940) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 107 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x40013c6e60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 108 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x40013c6e60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 109 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x40013b8350) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 110 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x40013c6e60, 0x4004519740) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 111 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001944c0, 0x1144559, 0x1b, 0x40000f5e38, 0x4004519860, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004ba1340, 0x4004519860, 0xf1f700, 0x4000195d40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40000f5f80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4002481bc0, 0x4004ba1340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 112 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 113 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x40013ccc30, 0xd00007, 0x4004bfe338, 0x5, 0x4004ba1350, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 130 [IO wait]: internal/poll.runtime_pollWait(0xffff68870400, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1818, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f1800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f1800, 0xffff92f32a68, 0x30, 0x400455fef0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004b97680, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004b97680, 0x400455fef0, 0x11ae500, 0x4002484940, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40024848c0, 0x12ed9d0, 0x4004b97680, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40024848c0, 0x12ed9d0, 0x4004b97680, 0x400014e380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 131 [IO wait]: internal/poll.runtime_pollWait(0xffff68870318, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1918, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f1900, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f1900, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004b97698, 0xffff92f32a68, 0xffff685f9b28, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004b97698, 0x665e1c, 0x0, 0xfe5e60, 0x40023de5c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004b976b0, 0x400455ff80, 0x11ae500, 0x4002484a20, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40024849a0, 0x12ec710, 0x4004b976b0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40024849a0, 0x12ec710, 0x4004b976b0, 0x400014e380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 146 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40000e9180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 138 [IO wait]: internal/poll.runtime_pollWait(0xffff686cda00, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1c98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f1c80, 0x4004bc3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f1c80, 0x4004bc3000, 0x1000, 0x1000, 0x40013f2be0, 0x469b10, 0x4004d84a80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40013b83e8, 0x4004bc3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004bfc480, 0x4004bc3000, 0x1000, 0x1000, 0x40013f2cf8, 0x67867c, 0x4004dce000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004d866c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004d866c0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004bfc480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 139 [select]: net/http.(*persistConn).writeLoop(0x4004bfc480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 129 [semacquire]: sync.runtime_Semacquire(0x4004e450e8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004e450e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x40000d2100, 0x40003727e0, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffff05e969, 0x29, 0xffffff05e9cb, 0x25, 0x4004c6f740, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004761e60, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffff05e969, 0x29, 0xffffff05e9cb, 0x25, 0x4004c6f740, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004761e60, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x400236b190, 0x400236ae60, 0x40003727e0, 0x400011f200, 0x40001f6ac0, 0x0, 0xbf, 0x40024b4da8, 0xffff686c8fe0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x400014e380, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004761e60, 0xb, 0x4004761e58, 0x6, 0x4004761e70, 0xd, 0x4004c0a0e0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x400014e380, 0x4000330000, 0x4000118030) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x400014e380, 0x400031aaf0, 0x2, 0x2, 0x400031ab00, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 86 [IO wait]: internal/poll.runtime_pollWait(0xffff688707a0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0398, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0380, 0x40013e9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0380, 0x40013e9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40013b8060, 0x40013e9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400011c900, 0x40013e9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000149440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000149440, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000149440, 0x41ab8c, 0xffff68441600, 0x100100000000, 0xffff685d8fff, 0x400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40045dc6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40045dc6c0, 0x4004d08400, 0x200, 0x200, 0x40023b3508, 0x41cff4, 0xffff68441600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000081940, 0x4004d08400, 0x200, 0x200, 0xffff683a63b0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000081940, 0x4004d08400, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000081980, 0x4004d08400, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004d0c000, 0x0, 0x40023b36b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004d0c000, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004d0c000, 0xee0d40, 0x400008cff0, 0x12db068, 0x4000081980) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004d062d0, 0x40045f4480, 0x4004d0a180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 147 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40000e9180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 148 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40000e9180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 149 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40000e9180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 150 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4004776bc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 151 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40047073f0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 152 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40047073f0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 153 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x40047073f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 154 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40000e9180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 155 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001944c0, 0x1144559, 0x1b, 0x4004e11e38, 0x4004c6e600, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004c1e5d0, 0x4004c6e600, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004e11f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004c72300, 0x4004c1e5d0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 160 [IO wait]: internal/poll.runtime_pollWait(0xffff686cdae8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400477e518, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400477e500, 0x4004c23000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400477e500, 0x4004c23000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000122588, 0x4004c23000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004c0c480, 0x4004c23000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004c72e40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004c72e40, 0xa, 0x0, 0x0, 0x0, 0x0, 0x4000378438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004c72e40, 0x0, 0x4000378478, 0x655784, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004c097a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004c097a0, 0x400477bc00, 0x200, 0x200, 0x4, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004776d00, 0x400477bc00, 0x200, 0x200, 0x400467db07, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004776d00, 0x400477bc00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004776d40, 0x400477bc00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004c76840, 0x0, 0x40003786b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004c76840, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004c76840, 0xee0d40, 0x40001513b0, 0x12db068, 0x4004776d40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004c1cfc0, 0x4004c72fc0, 0x4004c6f140) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 158 [select]: net/http.(*persistConn).readLoop(0x4004c0c480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 159 [select]: net/http.(*persistConn).writeLoop(0x4004c0c480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 161 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4004776c00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 162 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x4004707290) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 163 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x4004707290) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 164 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x4004707290) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 165 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x4004707290) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 166 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004c6cc00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 167 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x4004c6cc00, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 168 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x400014e380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 169 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x400014e380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 170 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x400014e380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 171 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:49:41.813+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:49:41.814+00:00 [Info] Setting ipv6=false 2021-08-16T13:49:41.821+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:49:41.825+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:49:41.825+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:49:41.825+00:00 [Info] Setting log level to Info 2021-08-16T13:49:41.825+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:49:41.828+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:49:41 audit: created new audit service 2021-08-16T13:49:41.836+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:49:41.836+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:49:41.836+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:49:41.836+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:49:41.836+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:49:41.849+00:00 [Info] security context initialized 2021-08-16T13:49:41.849+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:49:41.849+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:49:41.849+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:49:41.849+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:49:41.850+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:49:41.863+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:49:41.863+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:49:41.870+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:49:41.886+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:49:41.886+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:49:41.889+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:49:41.891+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:49:41.904+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:49:41.906+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:49:41.906+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:49:41.906+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:49:41.946+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:49:41.946+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 50 us 2021-08-16T13:49:41.946+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:49:41.948+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:49:41.949+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:49:41.951+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:49:41.951+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:49:41.951+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:49:41.951+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:49:41.951+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:49:41.953+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:41.953+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:41.954+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:49:41.954+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:49:41.954+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:49:41.954+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:41.954+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:49:41.954+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:49:41.954+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:49:41.954+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:49:41.954+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:49:41.954+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:49:41.954+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:49:41.961+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 6.586273ms 2021-08-16T13:49:41.961+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:49:41.962+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:49:41.962+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:41.962+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:49:41.962+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:49:41.962+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:49:41.962+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... unexpected fault address 0x4000237d5f 2021-08-16T13:49:41.977+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:49:41.977+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:49:41.977+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:49:41.978+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:49:41.978+00:00 [Info] : Recover shard shard1 from metadata completed. fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x4000237d5f pc=0xbd8ea4] goroutine 187 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004c7e9f0 sp=0x4004c7e9c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004c7ea30 sp=0x4004c7e9f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x4004e88340, 0x4004eb0000, 0x11a9890, 0x4004c7eb40, 0x4004e88358, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004c7eab0 sp=0x4004c7ea40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x4004e88340, 0x4004eb0000, 0x11a9890, 0x4004c7eb40, 0x4004e88358, 0xffff81a41280, 0x4004c7ec18, 0x49) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004c7eaf0 sp=0x4004c7eab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4000140b40, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004c7eda0 sp=0x4004c7eaf0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4000140b40, 0x400008d500, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004c7ee60 sp=0x4004c7eda0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x40046331c0, 0x40046331d0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004c7f200 sp=0x4004c7ee60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x40046331c0, 0x40046331d0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004c7f8f0 sp=0x4004c7f200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004e8af20, 0x4004e8c700, 0x40000edb80, 0x40046331a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004c7ffb0 sp=0x4004c7f8f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004c7ffb0 sp=0x4004c7ffb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x40002e4380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x400019d410, 0x4, 0x1136a87, 0xf, 0x400028be68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 20 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400013c660) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 21 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400013c678) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 22 [IO wait]: internal/poll.runtime_pollWait(0xffff81a42e88, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400029e098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400029e080, 0x40002b9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400029e080, 0x40002b9000, 0x1000, 0x1000, 0x1311860, 0x47b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000290008, 0x40002b9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x4000280120, 0x40002ce001, 0xdff, 0xdff, 0x400012f8e8, 0x6902d8, 0x4000287948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400028c048, 0x40002ce001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40002b6160, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40002b6160, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40002b6160, 0xf5a960, 0x4000280200, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x40002801e0, 0x4000292100, 0x4000287b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x40002c8000, 0x12f8f98, 0x40002801e0, 0x400029c8a7, 0x16, 0x40002ac360, 0x400012fc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x40002c8000, 0x12f8f98, 0x40002801e0, 0x40002c8000, 0x400029c0b0, 0x400029c0c0, 0x400029e100, 0x4000292020, 0xee0980, 0x40002ac360, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x40002c8000, 0x12f8f98, 0x40002801e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x400019c390, 0x400012ff78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400012bf78, 0x400019c390, 0x12db8a8, 0x400028c018, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x400019c390, 0x400014d220, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x400019c390, 0x400014d220) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 35 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x40002c2100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 23 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 24 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 25 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 26 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 27 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 28 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40004c67a0, 0x4, 0x40004c6668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40004c67a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x400010e060, 0x40004c67a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x400014a000, 0x40004c67a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 8 [IO wait]: internal/poll.runtime_pollWait(0xffff81a42da0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400029e218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400029e200, 0x40002cf000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400029e200, 0x40002cf000, 0x1000, 0x1000, 0x4000066a88, 0x68f7a4, 0x40002b6310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000290038, 0x40002cf000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40002ac000, 0x40002cf000, 0x1000, 0x1000, 0xf12180, 0x40002c8310, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000280900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000280900, 0xa, 0x1008880, 0x40002c82d0, 0x199, 0x1000, 0x4000066b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000280900, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000282330) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000282330, 0x4000090000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40002ba0c0, 0x4000090000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40002ba0c0, 0x4000090000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40002ba100, 0x4000090000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40002b62c0, 0x0, 0x4000066eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40002b62c0, 0x0, 0x0, 0x4000066f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40002b62c0, 0xee0d40, 0x40002c82d0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4000186000, 0x4000280a80, 0x40002a4a20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 6 [select]: net/http.(*persistConn).readLoop(0x40002ac000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 7 [select]: net/http.(*persistConn).writeLoop(0x40002ac000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 39 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001b83a0, 0x12c18bc, 0x1, 0x400012de58, 0x40002a4720, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x40004c36c8, 0x40002a4720, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400012dfa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x40002ba1c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 40 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x40002ba1c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 59 [select]: net/http.(*persistConn).readLoop(0x400047e5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 43 [IO wait]: internal/poll.runtime_pollWait(0xffff81a42cb8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000198098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000198080, 0x4000012000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000198080, 0x4000012000, 0x1000, 0x1000, 0x400029bbe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400014a368, 0x4000012000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c480, 0x4000012000, 0x1000, 0x1000, 0x400029bcf8, 0x67867c, 0x400011a7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400044d6e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x400044d6e0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000c480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 44 [select]: net/http.(*persistConn).writeLoop(0x400000c480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 47 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000494540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 60 [select]: net/http.(*persistConn).writeLoop(0x400047e5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 61 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000169cc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 102 [IO wait]: internal/poll.runtime_pollWait(0xffff81a422c0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000199018, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000199000, 0x4004549000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000199000, 0x4004549000, 0x1000, 0x1000, 0x40004e3688, 0x69f178, 0x40004e36f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400014a578, 0x4004549000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c7e0, 0x4004549000, 0x1000, 0x1000, 0x5, 0x4004414f00, 0x40004e37c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40024c9500) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40024c9500, 0x68f90a, 0x40004f24d0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40024c9500, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400019c900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400019c900, 0x40000aa000, 0x1000, 0x1000, 0x40004e3998, 0x44e884, 0x40004e3948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000169900, 0x40000aa000, 0x1000, 0x1000, 0x400045e4d8, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000169900, 0x40000aa000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000169940, 0x40000aa000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40004e3d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40004e3d80, 0xa, 0x40000aa201, 0x1, 0xffff586570a8, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40004e3d80, 0x40000aa20a, 0x1, 0x40044e4750, 0x0, 0x40000aa259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40004e3d80, 0x400452560a, 0x40044e4750, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40024faba0, 0x4004542ee0, 0x1f, 0x40004e3ee0, 0x40045414b8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x40024faba0, 0x113075a, 0x7, 0x40004966b0, 0xd, 0x40045414b8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x40024fab60, 0x40004966b0, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 165 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40000f07e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 49 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x400007e460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 62 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x400024ef00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 124 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 52 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x400007e460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 53 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x400007e460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 54 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x400007e460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 123 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001b83a0, 0x1144559, 0x1b, 0x40044ffe38, 0x40024f9500, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4002485570, 0x40024f9500, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x40024e9140) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40044fff80, 0x24, 0x100000001) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40024c9a40, 0x4002485570) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 70 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400041ee00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 114 [IO wait]: internal/poll.runtime_pollWait(0xffff81a42490, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400024c418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400024c400, 0x400447d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400024c400, 0x400447d000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40002400d0, 0x400447d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000156d80, 0x400447d000, 0x1000, 0x1000, 0x4000103200, 0x200000003, 0x4000103200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000227320) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000227320, 0x40004ec00a, 0x400012f8c8, 0x4d3b10, 0x400010e0c0, 0x40004ec000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000227320, 0x0, 0x1, 0x0, 0x400010e0d8, 0x40004ec000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40024e8f30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40024e8f30, 0x4004548000, 0x1000, 0x1000, 0x400012f9b8, 0x44d8a8, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400041e880, 0x4004548000, 0x1000, 0x1000, 0x40024c8d18, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400041e880, 0x4004548000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400041e8c0, 0x4004548000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400012fdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400012fdd0, 0xa, 0x4004548201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400012fdd0, 0x400454820a, 0x1, 0x4000036c0b, 0x0, 0x4004548263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400012fdd0, 0x400029240a, 0x4000036c0b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40024faba0, 0x4000164690, 0x24, 0x400012ff20, 0x40045411d0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x40024faba0, 0x113075a, 0x7, 0x40045411d0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x40024fab60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 33 [IO wait]: internal/poll.runtime_pollWait(0xffff81a423a8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000198298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000198280, 0x4004546000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000198280, 0x4004546000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400014a558, 0x4004546000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c6c0, 0x4004546000, 0x1000, 0x1000, 0x4000252000, 0x200000003, 0x4000252000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40024c9140) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40024c9140, 0x40004ec00a, 0x40001318c8, 0x4d3b10, 0x400010e0c0, 0x40004ec000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40024c9140, 0x0, 0x1, 0x2, 0x40001318d8, 0x41cd4c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000126d50) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000126d50, 0x40044f6000, 0x1000, 0x1000, 0x40001319b8, 0x0, 0x4000131988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000168100, 0x40044f6000, 0x1000, 0x1000, 0x40004de300, 0x3, 0x4000131be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000168100, 0x40044f6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000168580, 0x40044f6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000131dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000131dd0, 0xa, 0x40044f6201, 0x1, 0xd1f, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000131dd0, 0x40044f620a, 0x1, 0x4000496699, 0x0, 0x40044f62e1, 0x1, 0xd1f, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000131dd0, 0x4000238d0a, 0x4000496699, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40024faba0, 0x40002c6750, 0x17, 0x4000131f20, 0x40045411e8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x40024faba0, 0x113075a, 0x7, 0x40045411e8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x40024fab60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 164 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40000f07e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 71 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x40002269c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 72 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x4000216ab0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 73 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x400024c180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 74 [IO wait]: internal/poll.runtime_pollWait(0xffff81a42748, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400024c298, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400024c280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400024c280, 0x0, 0x400024a698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004478ea0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004478ea0, 0x12ed9d0, 0x4004478ea0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x400024c200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 75 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40002389c0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 76 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40002389c0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 77 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x40002389c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 78 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x4000217590) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 84 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000220d80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 82 [select]: net/http.(*persistConn).readLoop(0x4000156d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 81 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40001aa000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 83 [select]: net/http.(*persistConn).writeLoop(0x4000156d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 125 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x4000118960, 0xd00007, 0x40000372d0, 0x5, 0x4002485580, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 117 [select]: net/http.(*persistConn).readLoop(0x400000c6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 118 [select]: net/http.(*persistConn).writeLoop(0x400000c6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 121 [select]: net/http.(*persistConn).readLoop(0x400000c7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 122 [select]: net/http.(*persistConn).writeLoop(0x400000c7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 107 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40002ba780) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 105 [IO wait]: internal/poll.runtime_pollWait(0xffff81a421d8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400024c618, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400024c600, 0x4000096000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400024c600, 0x4000096000, 0x1000, 0x1000, 0x4000294be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000290080, 0x4000096000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4002552900, 0x4000096000, 0x1000, 0x1000, 0x4000294cf8, 0x67867c, 0x400011a840) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400010ec00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x400010ec00, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4002552900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 106 [select]: net/http.(*persistConn).writeLoop(0x4002552900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 85 [IO wait]: internal/poll.runtime_pollWait(0xffff81a42660, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400029ee18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400029ee00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400029ee00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c9a480, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c9a480, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x4004c94400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 86 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x4000221740, 0x40004f7490, 0x40004f7500, 0x40047d69e0, 0x12e8f90, 0x400442a620, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x4004bd71c4, 0x5, 0x4004c94400, 0x12eca70, 0x4004c942c0, 0x130d478, 0x400041c850, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40001907e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40001907e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 63 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x400024ef00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 64 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x4000221740, 0x40047d69e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 65 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x400442a620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 130 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x400442a620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 131 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x40044c62d0, 0x4004797560) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 132 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x40002ba740) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 133 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x400024efa0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 134 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x400024efa0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 135 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x4000240370) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 87 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x400024efa0, 0x40046c2ba0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 126 [IO wait]: internal/poll.runtime_pollWait(0xffff81a420f0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000199498, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000199480, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000199480, 0xffff81afea68, 0x30, 0x40046d3590) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004541938, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004541938, 0x40046d3590, 0x11ae500, 0x400007e5c0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x400007e540, 0x12ed9d0, 0x4004541938, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x400007e540, 0x12ed9d0, 0x4004541938, 0x40002e4380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 127 [IO wait]: internal/poll.runtime_pollWait(0xffff81a42008, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000199598, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000199580, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000199580, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004541950, 0xffff81afea68, 0xffff58657570, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004541950, 0x665e1c, 0x0, 0xfe5e60, 0x4000052dc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004541968, 0x40046d3620, 0x11ae500, 0x400007e6a0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x400007e620, 0x12ec710, 0x4004541968, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x400007e620, 0x12ec710, 0x4004541968, 0x40002e4380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 111 [IO wait]: internal/poll.runtime_pollWait(0xffff81a42578, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400045cb98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400045cb80, 0x400478c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400045cb80, 0x400478c000, 0x1000, 0x1000, 0x4000297a48, 0x40002e2a80, 0x40002e2a80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000240150, 0x400478c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400047e5a0, 0x400478c000, 0x1000, 0x1000, 0x0, 0x4000160dc0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000227bc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000227bc0, 0x100a, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000227bc0, 0x7356e4, 0xffffdfc2e94e, 0xe, 0x4004787ca8, 0x40b2f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40044ee690) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40044ee690, 0x40024ab000, 0x200, 0x200, 0x60, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40002bb340, 0x40024ab000, 0x200, 0x200, 0x40044f0de0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40002bb340, 0x40024ab000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40002bb380, 0x40024ab000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x400462ef20, 0x680d00, 0x4004787eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400462ef20, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400462ef20, 0xee0d40, 0x40002c8be0, 0x12db068, 0x40002bb380) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400462a750, 0x4004630300, 0x40044f0e40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 109 [IO wait]: internal/poll.runtime_pollWait(0xffff5832e450, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400024d198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400024d180, 0x4004636000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400024d180, 0x4004636000, 0x1000, 0x1000, 0x4000067be0, 0x469b10, 0x40002e3c80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000290318, 0x4004636000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004c367e0, 0x4004636000, 0x1000, 0x1000, 0x4000067cf8, 0x67867c, 0x40044f0de0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004630240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004630240, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004c367e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 110 [select]: net/http.(*persistConn).writeLoop(0x4004c367e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 163 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40000f07e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 162 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40000f07e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 113 [semacquire]: sync.runtime_Semacquire(0x4004e8af28) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004e8af20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x40000edb80, 0x400048e6f0, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffdfc2e969, 0x29, 0xffffdfc2e9cb, 0x25, 0x400464a9c0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004bd7eb0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffdfc2e969, 0x29, 0xffffdfc2e9cb, 0x25, 0x400464a9c0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004bd7eb0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004c7f190, 0x4004c7ee60, 0x400048e6f0, 0x40004ba6c0, 0x4000494540, 0x0, 0xbf, 0x40044fcda8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x40002e4380, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004bd7eb0, 0xb, 0x4004bd7ea8, 0x6, 0x4004bd7ec0, 0xd, 0x4004c92720, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x40002e4380, 0x4000440000, 0x40002829f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x40002e4380, 0x4000228c30, 0x2, 0x2, 0x4000228c40, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 147 [IO wait]: internal/poll.runtime_pollWait(0xffff5832e538, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400024d398, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400024d380, 0x4004641000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400024d380, 0x4004641000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000290400, 0x4004641000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40025538c0, 0x4004641000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004631140) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004631140, 0xa, 0x4004520123, 0x11320c7, 0x9, 0x597c01, 0x40002474b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004631140, 0xd4fa28, 0x4000144000, 0x46, 0x11320c7, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004e1c450) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004e1c450, 0x4004529a00, 0x200, 0x200, 0x4000037370, 0x4000037370, 0x40045262d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400041f3c0, 0x4004529a00, 0x200, 0x200, 0x40002aab47, 0xe, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400041f3c0, 0x4004529a00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400041f400, 0x4004529a00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004e342c0, 0x40024f9500, 0x40002476b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004e342c0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004e342c0, 0xee0d40, 0x4000118d70, 0x12db068, 0x400041f400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004e1e3f0, 0x4004e20120, 0x4004e2e240) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 166 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x40002bb780) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 167 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40004a1340, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 168 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40004a1340, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 169 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x40004a1340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 170 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40000f07e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 171 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001b83a0, 0x1144559, 0x1b, 0x4004cdfe38, 0x40044f1620, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004632820, 0x40044f1620, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004cdff80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004630780, 0x4004632820) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 176 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40002bb7c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 174 [select]: net/http.(*persistConn).readLoop(0x40025538c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 175 [select]: net/http.(*persistConn).writeLoop(0x40025538c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 177 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x40004a11e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 178 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x40004a11e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 179 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x40004a11e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 180 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x40004a11e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 181 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004658500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 182 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x4004658500, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 183 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x40002e4380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 184 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x40002e4380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 185 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x40002e4380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 186 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:49:47.140+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:49:47.140+00:00 [Info] Setting ipv6=false 2021-08-16T13:49:47.149+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:49:47.153+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:49:47.153+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:49:47.153+00:00 [Info] Setting log level to Info 2021-08-16T13:49:47.154+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:49:47.156+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:49:47 audit: created new audit service 2021-08-16T13:49:47.164+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:49:47.164+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:49:47.164+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:49:47.165+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:49:47.165+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:49:47.177+00:00 [Info] security context initialized 2021-08-16T13:49:47.177+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:49:47.177+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:49:47.178+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:49:47.178+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:49:47.178+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:49:47.202+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:49:47.207+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:49:47.219+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:49:47.232+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:49:47.234+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:49:47.243+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:49:47.243+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:49:47.244+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:49:47.244+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:49:47.246+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:49:47.254+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:49:47.318+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:49:47.318+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 54 us 2021-08-16T13:49:47.318+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:49:47.320+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:49:47.320+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:49:47.324+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:49:47.324+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:49:47.324+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:49:47.324+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:49:47.324+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:49:47.324+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:47.324+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:47.325+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:49:47.325+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:49:47.325+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:49:47.325+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:47.326+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:49:47.326+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:49:47.326+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:49:47.326+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:49:47.326+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:49:47.326+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:49:47.326+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:49:47.333+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 6.865237ms 2021-08-16T13:49:47.333+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:49:47.334+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:49:47.334+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:47.334+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:49:47.334+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:49:47.334+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:49:47.334+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:49:47.354+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:49:47.354+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:49:47.354+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:49:47.354+00:00 [Info] RebalanceMgr::initService Init 2021-08-16T13:49:47.354+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:49:47.354+00:00 [Info] ServiceMgr::registerWithServer nodeuuid 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:47.354+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x40001cb21f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x40001cb21f pc=0xbd8ea4] goroutine 170 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x40034ee9f0 sp=0x40034ee9c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x40034eea30 sp=0x40034ee9f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x40001571e0, 0x4004eb4000, 0x11a9890, 0x40034eeb40, 0x40001571f8, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x40034eeab0 sp=0x40034eea40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x40001571e0, 0x4004eb4000, 0x11a9890, 0x40034eeb40, 0x40001571f8, 0xc18f8c, 0x40034eec18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x40034eeaf0 sp=0x40034eeab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4000158780, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x40034eeda0 sp=0x40034eeaf0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4000158780, 0x400007f490, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x40034eee60 sp=0x40034eeda0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004cc7550, 0x4004cc7560, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x40034ef200 sp=0x40034eee60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004cc7550, 0x4004cc7560, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x40034ef8f0 sp=0x40034ef200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004ea8420, 0x4004ea0380, 0x4004cd2b00, 0x4004cc7530) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x40034effb0 sp=0x40034ef8f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x40034effb0 sp=0x40034effb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x4000168380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x400008d770, 0x4, 0x1136a87, 0xf, 0x4000297e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 5 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400000e120) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 6 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400000e138) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 7 [IO wait]: internal/poll.runtime_pollWait(0xffff8c8f3758, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000208118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000208100, 0x4000211000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000208100, 0x4000211000, 0x1000, 0x1000, 0x1311860, 0x47b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b84e8, 0x4000211000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x40001ba360, 0x400013c001, 0xdff, 0xdff, 0x400019f8e8, 0x6902d8, 0x4000293948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x40001aa588, 0x400013c001, 0xdff, 0xdff, 0x40002939f8, 0x6f1968, 0x4000293938) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40001e66e0, 0xa, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40001e66e0, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40001e66e0, 0xf5a960, 0x40001ba440, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x40001ba420, 0x40001200e0, 0x4000293b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x40001801e0, 0x12f8f98, 0x40001ba420, 0x40001367d7, 0x16, 0x4000142120, 0x400019fc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x40001801e0, 0x12f8f98, 0x40001ba420, 0x40001801e0, 0x40001b4490, 0x40001b44a0, 0x4000208180, 0x40001c21e0, 0xee0980, 0x4000142120, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x40001801e0, 0x12f8f98, 0x40001ba420) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x400008c6f0, 0x400019ff78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x40004f1f78, 0x400008c6f0, 0x12db8a8, 0x40001aa558, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x400008c6f0, 0x400009c460, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x400008c6f0, 0x400009c460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 8 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 9 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 10 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 11 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 12 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 34 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400010c200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 13 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40004ac7a0, 0x4, 0x40004ac668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40004ac7a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x40001ba000, 0x40004ac7a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x40001b8000, 0x40004ac7a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 56 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000123100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 50 [select]: net/http.(*persistConn).readLoop(0x4000142a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 51 [select]: net/http.(*persistConn).writeLoop(0x4000142a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 40 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400006c790, 0x12c18bc, 0x1, 0x400019be58, 0x400016a6c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x4000019ec8, 0x400016a6c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400019bfa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4000122140) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 41 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4000122140) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 46 [IO wait]: internal/poll.runtime_pollWait(0xffff8c8f3670, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000166198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000166180, 0x400013d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000166180, 0x400013d000, 0x1000, 0x1000, 0x4000283a88, 0x68f7a4, 0x40001e6890) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400029a018, 0x400013d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000142a20, 0x400013d000, 0x1000, 0x1000, 0xf12180, 0x4000094270, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400010e6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400010e6c0, 0xa, 0x1008880, 0x4000094230, 0x199, 0x1000, 0x4000283b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400010e6c0, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000492cf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000492cf0, 0x400001e000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000080200, 0x400001e000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000080200, 0x400001e000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000080240, 0x400001e000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40001e6840, 0x0, 0x4000283eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40001e6840, 0x0, 0x0, 0x4000283f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40001e6840, 0xee0d40, 0x4000094230, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4000140ea0, 0x40004633e0, 0x40001825a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 44 [IO wait]: internal/poll.runtime_pollWait(0xffff8c8f3588, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000208098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000208080, 0x400001c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000208080, 0x400001c000, 0x1000, 0x1000, 0x4000281be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011c1a8, 0x400001c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c66c0, 0x400001c000, 0x1000, 0x1000, 0x4000281cf8, 0x67867c, 0x4000182540) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000463260) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000463260, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40001c66c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 45 [select]: net/http.(*persistConn).writeLoop(0x40001c66c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 54 [select]: net/http.(*persistConn).readLoop(0x400000d560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 55 [select]: net/http.(*persistConn).writeLoop(0x400000d560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 62 [IO wait]: internal/poll.runtime_pollWait(0xffff8c8f3018, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400008ef18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400008ef00, 0x40002bb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400008ef00, 0x40002bb000, 0x1000, 0x1000, 0x4000297688, 0x69f178, 0x40002976f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400029a290, 0x40002bb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000d560, 0x40002bb000, 0x1000, 0x1000, 0x5, 0x4000258000, 0x40002977c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400010f3e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400010f3e0, 0x68f90a, 0x40002222c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400010f3e0, 0x0, 0x4000297868, 0x41a0a8, 0x4000297888, 0x41cd4c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000224270) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000224270, 0x4000256000, 0x1000, 0x1000, 0x4000297998, 0x44e884, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400022c140, 0x4000256000, 0x1000, 0x1000, 0x400008a598, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400022c140, 0x4000256000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400022c180, 0x4000256000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000297d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000297d80, 0xa, 0x4000256201, 0x1, 0xffff8c8f8ba8, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000297d80, 0x400025620a, 0x1, 0x40044da0a0, 0x0, 0x4000256259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000297d80, 0x400026600a, 0x40044da0a0, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40004fc6c0, 0x400257c760, 0x1f, 0x4000297ee0, 0x40024cd788, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x40004fc6c0, 0x113075a, 0x7, 0x400029dc20, 0xd, 0x40024cd788, 0x0, 0x0, 0x2) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x40004fc680, 0x400029dc20, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 67 [IO wait]: internal/poll.runtime_pollWait(0xffff8c8f2e48, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000208518, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000208500, 0x400020b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000208500, 0x400020b000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400029a0a8, 0x400020b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c6c60, 0x400020b000, 0x1000, 0x1000, 0x40004aed80, 0x200000003, 0x40004aed80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001bac60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001bac60, 0x40002a800a, 0x40004f58c8, 0x4d3b10, 0x40001ba060, 0x40002a8000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40001bac60, 0x0, 0x1, 0x0, 0x40001ba078, 0x40002a8000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4002516b10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4002516b10, 0x4000213000, 0x1000, 0x1000, 0x40004f59b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000122600, 0x4000213000, 0x1000, 0x1000, 0x40001ba7d8, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000122600, 0x4000213000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000122640, 0x4000213000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40004f5dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40004f5dd0, 0xa, 0x4000213201, 0x1, 0xd1f, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40004f5dd0, 0x400021320a, 0x1, 0x400029dc09, 0x0, 0x40002132e1, 0x1, 0xd1f, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40004f5dd0, 0x400019730a, 0x400029dc09, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40004fc6c0, 0x400003ab40, 0x17, 0x40004f5f20, 0x4002514750, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x40004fc6c0, 0x113075a, 0x7, 0x4002514750, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x40004fc680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 115 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400022cdc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 116 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40002f4800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 15 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x40034f4000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 26 [select]: net/http.(*persistConn).writeLoop(0x4000246240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 86 [select]: net/http.(*persistConn).readLoop(0x4000142ea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 83 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x40034f4000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 84 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x40034f4000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 85 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x40034f4000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 87 [select]: net/http.(*persistConn).writeLoop(0x4000142ea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 68 [IO wait]: internal/poll.runtime_pollWait(0xffff8c8f2f30, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000166598, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000166580, 0x40034f6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000166580, 0x40034f6000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010168, 0x40034f6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000142ea0, 0x40034f6000, 0x1000, 0x1000, 0x40004af200, 0x200000003, 0x40004af200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000288ea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000288ea0, 0x40002a800a, 0x40004f38c8, 0x4d3b10, 0x40001ba060, 0x40002a8000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000288ea0, 0x0, 0x1, 0x0, 0x40001ba078, 0x40002a8000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400028b140) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400028b140, 0x400257e000, 0x1000, 0x1000, 0x40004f39b8, 0x0, 0x40004f3988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40001d9a80, 0x400257e000, 0x1000, 0x1000, 0x4000102f00, 0x3, 0x40004f3be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40001d9a80, 0x400257e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001d9ac0, 0x400257e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40004f3dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40004f3dd0, 0xa, 0x400257e201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40004f3dd0, 0x400257e20a, 0x1, 0x400029da2b, 0x0, 0x400257e263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40004f3dd0, 0x40001c240a, 0x400029da2b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40004fc6c0, 0x400015a630, 0x24, 0x40004f3f20, 0x4002514738, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x40004fc6c0, 0x113075a, 0x7, 0x4002514738, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x40004fc680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 73 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001d9780) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 71 [select]: net/http.(*persistConn).readLoop(0x40001c6c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 72 [select]: net/http.(*persistConn).writeLoop(0x40001c6c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 74 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x40001bb0e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 75 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x4002516e10) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 76 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x4000208580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 77 [IO wait]: internal/poll.runtime_pollWait(0xffff8c8f2c78, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000208698, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000208680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000208680, 0x0, 0x40004a9e98, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4002514bd0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4002514bd0, 0x12ed9d0, 0x4002514bd0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x4000208600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 78 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40004fcb60, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 79 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40004fcb60, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 80 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x40004fcb60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 81 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x40025178f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 144 [semacquire]: sync.runtime_Semacquire(0x4004ea8428) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004ea8420) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x4004cd2b00, 0x4000492840, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffe9a3d969, 0x29, 0xffffe9a3d9cb, 0x25, 0x4004e981e0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x400021f8a0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffe9a3d969, 0x29, 0xffffe9a3d9cb, 0x25, 0x4004e981e0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x400021f8a0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x40034ef190, 0x40034eee60, 0x4000492840, 0x40000d5b00, 0x4000123100, 0x0, 0xffffb67c0130, 0x4004cecda8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x4000168380, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x400021f8a0, 0xb, 0x400021f898, 0x6, 0x400021f8b0, 0xd, 0x40001e8420, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x4000168380, 0x4000460000, 0x400013e720) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x4000168380, 0x4002518b40, 0x2, 0x2, 0x4002518b50, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 29 [select]: net/http.(*persistConn).writeLoop(0x40002df9e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 27 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400022c3c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 117 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x40004ff360) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 145 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x4000200a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 102 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x400017e150) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 25 [select]: net/http.(*persistConn).readLoop(0x4000246240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 90 [IO wait]: internal/poll.runtime_pollWait(0xffff8c8f2d60, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000166d98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000166d80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000166d80, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40024d4000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40024d4000, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x4000080f80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 91 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x40002f5240, 0x400007e770, 0x400007e7e0, 0x4004bcc630, 0x12e8f90, 0x40034f4620, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x4004b91664, 0x5, 0x4000080f80, 0x12eca70, 0x40002f5080, 0x130d478, 0x40001f82a0, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40034f4b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40034f4b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 118 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x40004ff360) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 119 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x40002f5240, 0x4004bcc630) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 92 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40034f4620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 93 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40034f4620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 94 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x40000ab440, 0x4000183980) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 95 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x400022c380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 96 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x40001bd2c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 97 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x40001bd2c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 130 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x400011c418) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 131 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x40001bd2c0, 0x40001839e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 132 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400006c790, 0x1144559, 0x1b, 0x4000291e38, 0x4000183b00, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004cc6090, 0x4000183b00, 0xf1f700, 0x40004e0010) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000291f80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40024575c0, 0x4004cc6090) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 133 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 134 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x40001808c0, 0xd00007, 0x4004cac7e8, 0x5, 0x4004cc60a0, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 135 [runnable]: internal/poll.runtime_pollWait(0xffff8c8f29c0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000167098, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000167080, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000167080, 0xffffb67c0108, 0x30, 0x40024d9260) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40024d4888, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40024d4888, 0x40024d9260, 0x11ae500, 0x40002006a0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x4000200620, 0x12ed9d0, 0x40024d4888, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x4000200620, 0x12ed9d0, 0x40024d4888, 0x4000168380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 136 [IO wait]: internal/poll.runtime_pollWait(0xffff8c8f28d8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000167198, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000167180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000167180, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40024d48a0, 0xffffb67c0108, 0xffff8c8ab328, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40024d48a0, 0x665e1c, 0x0, 0xfe5e60, 0x400243f5c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x40024d48b8, 0x40024d92f0, 0x11ae500, 0x4000200780, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x4000200700, 0x12ec710, 0x40024d48b8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x4000200700, 0x12ec710, 0x40024d48b8, 0x4000168380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 28 [IO wait]: internal/poll.runtime_pollWait(0xffff8c19b7a8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000167418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000167400, 0x4004b3a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000167400, 0x4004b3a000, 0x1000, 0x1000, 0x40002b3be0, 0x469b10, 0x40024c7c80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b8698, 0x4004b3a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40002df9e0, 0x4004b3a000, 0x1000, 0x1000, 0x40002b3cf8, 0x67867c, 0x40044f3e60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40046b4d20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40046b4d20, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40002df9e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 147 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x4000200a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 143 [IO wait]: internal/poll.runtime_pollWait(0xffff8c8f2b90, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4002508118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4002508100, 0x4002429000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4002508100, 0x4002429000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b8118, 0x4002429000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000246240, 0x4002429000, 0x1000, 0x1000, 0x0, 0x0, 0x40002b2b28) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400010e540) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400010e540, 0xa, 0x0, 0x0, 0xffff8c8ab328, 0x0, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400010e540, 0x413b8c, 0x30, 0x1063520, 0x400243ec01, 0x4ad6e4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40024d9d70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40024d9d70, 0x4004d04600, 0x200, 0x200, 0x4000184400, 0x4000081740, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000081ac0, 0x4004d04600, 0x200, 0x200, 0x4000200620, 0x4000081740, 0x1135447) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000081ac0, 0x4004d04600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000081b00, 0x4004d04600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004ccadc0, 0x12ed900, 0x400243eeb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004ccadc0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004ccadc0, 0xee0d40, 0x4000180f50, 0x12db068, 0x4000081b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004cb6b40, 0x4004d00300, 0x4004d023c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 122 [IO wait]: internal/poll.runtime_pollWait(0xffff8c19b6c0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400008fd98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400008fd80, 0x4004dc8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400008fd80, 0x4004dc8000, 0x1000, 0x1000, 0x4004cbdbe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000103a0, 0x4004dc8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004bca7e0, 0x4004dc8000, 0x1000, 0x1000, 0x4004cbdcf8, 0x67867c, 0x4004669b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40046436e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40046436e0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004bca7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 123 [select]: net/http.(*persistConn).writeLoop(0x4004bca7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 146 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x4000200a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 148 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x4000200a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 149 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4000081f00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 150 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40001dd810, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 151 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40001dd810, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 152 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x40001dd810) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 153 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x4000200a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 154 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400006c790, 0x1144559, 0x1b, 0x40024abe38, 0x4004d02e40, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004cc6bb0, 0x4004d02e40, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40024abf80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004d006c0, 0x4004cc6bb0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 159 [select]: net/http.(*persistConn).roundTrip(0x4004bca7e0, 0x4000123c80, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x40045f8a00, 0xf8, 0x10a8660, 0x400019ee18) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x40045f8a00, 0x1b6e900, 0xc03eb93cd51f7d2b, 0x1bfda22ce2) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x40045f8900, 0x12db048, 0x1b6e900, 0xc03eb93cd51f7d2b, 0x1bfda22ce2, 0x1be4a80, 0x0, 0x400029a480, 0x40045fca50, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x40045fca20, 0x40045f8900, 0xc03eb93cd51f7d2b, 0x1bfda22ce2, 0x1be4a80, 0x400029a480, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x40045fca20, 0x40045f8900, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004e50120, 0x1b, 0x400019f580, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x40002cb950, 0x112f905, 0x6, 0x12db908, 0x4000047100, 0xee40c0, 0x4000208b98, 0x400019f580, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x4004caa300, 0x46, 0x12db908, 0x4000047100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x4004caa300, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x20, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400019fd40, 0x4, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x40002498c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x40002498c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000081f40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1267 +0xf8 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 157 [select]: net/http.(*persistConn).readLoop(0x40002dfc20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 158 [select]: net/http.(*persistConn).writeLoop(0x40002dfc20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 106 [IO wait]: internal/poll.runtime_pollWait(0xffff8c8f2aa8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000167a98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000167a80, 0x4004d4e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000167a80, 0x4004d4e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011c5e0, 0x4004d4e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40002dfc20, 0x4004d4e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004d01020) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004d01020, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004d01020, 0x41ab8c, 0xffff8c498d00, 0x100100000000, 0xffff8c668fff, 0x400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40045fc300) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40045fc300, 0x4004e0a200, 0x200, 0x200, 0x4000019508, 0x41cff4, 0xffff8c498d00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40001238c0, 0x4004e0a200, 0x200, 0x200, 0xffff8c8f7460, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40001238c0, 0x4004e0a200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000123900, 0x4004e0a200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40045fa840, 0x0, 0x40000196b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40045fa840, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40045fa840, 0xee0d40, 0x4000094d20, 0x12db068, 0x4000123900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40002cb050, 0x4004e08240, 0x4004588d80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 160 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x40001dd6b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:224 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 161 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x40001dd6b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:612 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 162 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x40001dd6b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:662 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 163 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x40001dd6b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:752 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 107 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).processSchedIndexes(0x40001dd6b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:296 +0x44 created by github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:218 +0x3c goroutine 164 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004d44b00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:249 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 112 [runnable]: internal/poll.runtime_pollWait(0xffff8c19b4f0, 0x77, 0x4004ccde00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000208e18, 0x77, 0x12f8600, 0x4004e09020, 0x4000208e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitWrite(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:96 internal/poll.(*FD).WaitWrite(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:528 net.(*netFD).connect(0x4000208e00, 0x12f8668, 0x4004e09020, 0x0, 0x0, 0x12db428, 0x4004e4cba0, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:141 +0x1e4 net.(*netFD).dial(0x4000208e00, 0x12f8668, 0x4004e09020, 0x12ff2c0, 0x0, 0x12ff2c0, 0x40045fce70, 0x0, 0xffff8c8f70a8, 0x10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:149 +0xac net.socket(0x12f8668, 0x4004e09020, 0x112eab2, 0x3, 0x2, 0x1, 0x0, 0x0, 0x12ff2c0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:70 +0x148 net.internetSocket(0x12f8668, 0x4004e09020, 0x112eab2, 0x3, 0x12ff2c0, 0x0, 0x12ff2c0, 0x40045fce70, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/ipsock_posix.go:141 +0xcc net.(*sysDialer).doDialTCP(0x4000208d80, 0x12f8668, 0x4004e09020, 0x0, 0x40045fce70, 0xf74f40, 0x1c1c218, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:65 +0x8c net.(*sysDialer).dialTCP(0x4000208d80, 0x12f8668, 0x4004e09020, 0x0, 0x40045fce70, 0x46e908, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:61 +0xb8 net.(*sysDialer).dialSingle(0x4000208d80, 0x12f8668, 0x4004e09020, 0x12e9300, 0x40045fce70, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:580 +0x4d4 net.(*sysDialer).dialSerial(0x4000208d80, 0x12f8668, 0x4004e09020, 0x4004e0e980, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:548 +0xf8 net.(*Dialer).DialContext(0x40001ba240, 0x12f8668, 0x4004e09020, 0x112eab2, 0x3, 0x4004e0d2f0, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:425 +0x550 net/http.(*Transport).dial(0x1b6e900, 0x12f8668, 0x4004e08d20, 0x112eab2, 0x3, 0x4004e0d2f0, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1162 +0x178 net/http.(*Transport).dialConn(0x1b6e900, 0x12f8668, 0x4004e08d20, 0x0, 0x4004e50360, 0x4, 0x4004e0d2f0, 0xe, 0x0, 0x400000cb40, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1600 +0x155c net/http.(*Transport).dialConnFor(0x1b6e900, 0x4004e4e2c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1442 +0x90 created by net/http.(*Transport).queueForDial /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1411 +0x3a8 goroutine 166 [select]: net/http.(*Transport).getConn(0x1b6e900, 0x4000123cc0, 0x0, 0x4004e50270, 0x4, 0x4004e0d290, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1368 +0x418 net/http.(*Transport).roundTrip(0x1b6e900, 0x40045f8c00, 0xf8, 0x10a8660, 0x4004e5edd8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:579 +0x5c4 net/http.(*Transport).RoundTrip(0x1b6e900, 0x40045f8c00, 0x1b6e900, 0xc03eb93cd520ca67, 0x1bfda37a1e) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x40045f8b00, 0x12db048, 0x1b6e900, 0xc03eb93cd520ca67, 0x1bfda37a1e, 0x1be4a80, 0x0, 0x400029a490, 0x40045fcc00, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x40045fcbd0, 0x40045f8b00, 0xc03eb93cd520ca67, 0x1bfda37a1e, 0x1be4a80, 0x400029a490, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x40045fcbd0, 0x40045f8b00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004e50240, 0x1b, 0x4004e5f540, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x40002cbd40, 0x112f905, 0x6, 0x12db908, 0x4000047160, 0xee40c0, 0x4000208c18, 0x4004e5f540, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x400003e380, 0x46, 0x12db908, 0x4000047160, 0x4004e07598, 0x81ee08, 0x4004e07748, 0x95dcbc, 0x4004caa300, 0x46, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x400003e380, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x240, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004e5fd00, 0x40045fcb10, 0xffff8c8ac290) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x40034d1440, 0xd47600, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x40034d1440, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x4000168380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9702 +0x344 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 167 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x4000168380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 168 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x4000168380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 108 [runnable]: internal/poll.runtime_pollWait(0xffff8c19b5d8, 0x77, 0x4004e54180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000208d18, 0x77, 0x12f8600, 0x4004e08c00, 0x4000208d00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitWrite(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:96 internal/poll.(*FD).WaitWrite(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:528 net.(*netFD).connect(0x4000208d00, 0x12f8668, 0x4004e08c00, 0x0, 0x0, 0x12db428, 0x4004e4cb40, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:141 +0x1e4 net.(*netFD).dial(0x4000208d00, 0x12f8668, 0x4004e08c00, 0x12ff2c0, 0x0, 0x12ff2c0, 0x40045fccf0, 0x0, 0xffff8c8f70a8, 0x10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:149 +0xac net.socket(0x12f8668, 0x4004e08c00, 0x112eab2, 0x3, 0x2, 0x1, 0x0, 0x0, 0x12ff2c0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:70 +0x148 net.internetSocket(0x12f8668, 0x4004e08c00, 0x112eab2, 0x3, 0x12ff2c0, 0x0, 0x12ff2c0, 0x40045fccf0, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/ipsock_posix.go:141 +0xcc net.(*sysDialer).doDialTCP(0x4000208c80, 0x12f8668, 0x4004e08c00, 0x0, 0x40045fccf0, 0xf74f40, 0x1c1c218, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:65 +0x8c net.(*sysDialer).dialTCP(0x4000208c80, 0x12f8668, 0x4004e08c00, 0x0, 0x40045fccf0, 0x46e908, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:61 +0xb8 net.(*sysDialer).dialSingle(0x4000208c80, 0x12f8668, 0x4004e08c00, 0x12e9300, 0x40045fccf0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:580 +0x4d4 net.(*sysDialer).dialSerial(0x4000208c80, 0x12f8668, 0x4004e08c00, 0x4004e0e870, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:548 +0xf8 net.(*Dialer).DialContext(0x40001ba240, 0x12f8668, 0x4004e08c00, 0x112eab2, 0x3, 0x4004e0d290, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:425 +0x550 net/http.(*Transport).dial(0x1b6e900, 0x12f8668, 0x4004e08900, 0x112eab2, 0x3, 0x4004e0d290, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1162 +0x178 net/http.(*Transport).dialConn(0x1b6e900, 0x12f8668, 0x4004e08900, 0x0, 0x4004e50270, 0x4, 0x4004e0d290, 0xe, 0x0, 0x400000c900, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1600 +0x155c net/http.(*Transport).dialConnFor(0x1b6e900, 0x4004e4e210) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1442 +0x90 created by net/http.(*Transport).queueForDial /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1411 +0x3a8 goroutine 169 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 goroutine 109 [select]: net.(*netFD).connect.func2(0x12f8668, 0x4004e08c00, 0x4000208d00, 0x40045899e0, 0x4004589980) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:118 +0x70 created by net.(*netFD).connect /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:117 +0x1c0 goroutine 110 [runnable]: net/url.(*Userinfo).String(0x40045fced0, 0x4004589ce0, 0x4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/url/url.go:423 +0xc8 net/url.(*URL).String(0x4004e60360, 0x1, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/url/url.go:825 +0x868 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x40045fcf00, 0x4004e75f10, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:124 +0x130 github.com/couchbase/cbauth/revrpc.BabysitService(0x4004e75f10, 0x40045fcf00, 0x0, 0x0, 0x0, 0xd8d03c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth/service.RegisterManager(0x13007d8, 0x4004d44b00, 0x0, 0x0, 0x4004e07f80, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:146 +0xa0 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).registerWithServer(0x4004d44b00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:301 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:205 +0x78 goroutine 111 [select]: net/http.(*Transport).getConn(0x1b6e900, 0x4000123d00, 0x0, 0x4004e50360, 0x4, 0x4004e0d2f0, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1368 +0x418 net/http.(*Transport).roundTrip(0x1b6e900, 0x40045f8e00, 0xf8, 0x10a8660, 0x4004e07798) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:579 +0x5c4 net/http.(*Transport).RoundTrip(0x1b6e900, 0x40045f8e00, 0x1b6e900, 0xc03eba4ad5229a68, 0x11772a97a1f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x40045f8d00, 0x12db048, 0x1b6e900, 0xc03eba4ad5229a68, 0x11772a97a1f, 0x1be4a80, 0x0, 0x400029a4a8, 0x40045fcd80, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x40045fcd50, 0x40045f8d00, 0xc03eba4ad5229a68, 0x11772a97a1f, 0x1be4a80, 0x400029a4a8, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x40045fcd50, 0x40045f8d00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004e4cb80, 0x1f, 0x4004e07d10, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/indexer.getWithAuth(0x4004e4cb80, 0x1f, 0xe, 0x113846e, 0x11) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:3077 +0x68 github.com/couchbase/indexing/secondary/indexer.getGlobalTopology(0x4004d4da40, 0xe, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:2266 +0x78 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).updateNodeList(0x4004d44b00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:220 +0x3c created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:206 +0x98 goroutine 113 [select]: net.(*netFD).connect.func2(0x12f8668, 0x4004e09020, 0x4000208e00, 0x4004589c80, 0x4004589c20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:118 +0x70 created by net.(*netFD).connect /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:117 +0x1c0 Initializing write barrier = 8000 2021-08-16T13:49:52.575+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:49:52.580+00:00 [Info] Setting ipv6=false 2021-08-16T13:49:52.591+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:49:52.597+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:49:52.597+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:49:52.597+00:00 [Info] Setting log level to Info 2021-08-16T13:49:52.597+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:49:52.599+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:49:52 audit: created new audit service 2021-08-16T13:49:52.609+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:49:52.609+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:49:52.609+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:49:52.609+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:49:52.609+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:49:52.624+00:00 [Info] security context initialized 2021-08-16T13:49:52.624+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:49:52.624+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:49:52.624+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:49:52.624+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:49:52.624+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:49:52.640+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:49:52.640+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:49:52.648+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:49:52.669+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:49:52.672+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:49:52.672+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:49:52.673+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:49:52.673+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:49:52.673+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:49:52.675+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:49:52.677+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:49:52.740+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:49:52.740+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 54 us 2021-08-16T13:49:52.741+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:49:52.744+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:49:52.744+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:49:52.745+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:49:52.745+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:49:52.745+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:49:52.745+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:49:52.745+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:49:52.749+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:52.749+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:52.751+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:49:52.751+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:49:52.751+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:49:52.751+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:52.751+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:49:52.751+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:49:52.751+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:49:52.751+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:49:52.751+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:49:52.752+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:49:52.752+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:49:52.756+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 4.920284ms 2021-08-16T13:49:52.757+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:49:52.758+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:49:52.758+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:52.758+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:49:52.758+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:49:52.758+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:49:52.758+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:49:52.770+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:49:52.770+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:49:52.771+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:49:52.771+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:49:52.771+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x4004e0249f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x4004e0249f pc=0xbd8ea4] goroutine 168 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x40044189f0 sp=0x40044189c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004418a30 sp=0x40044189f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x40023f9a00, 0x4004e56000, 0x11a9890, 0x4004418b40, 0x40023f9a18, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004418ab0 sp=0x4004418a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x40023f9a00, 0x4004e56000, 0x11a9890, 0x4004418b40, 0x40023f9a18, 0xc18f8c, 0x4004418c18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004418af0 sp=0x4004418ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4000033680, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004418da0 sp=0x4004418af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4000033680, 0x4000151420, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004418e60 sp=0x4004418da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004777a30, 0x4004777a40, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004419200 sp=0x4004418e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004777a30, 0x4004777a40, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x40044198f0 sp=0x4004419200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004e44080, 0x4004e2aa80, 0x400469b600, 0x4004777a10) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004419fb0 sp=0x40044198f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004419fb0 sp=0x4004419fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x400143a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x40000fb5c0, 0x4, 0x1136a87, 0xf, 0x40000f9e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 7 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400000e690) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 8 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400000e6a8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 9 [IO wait]: internal/poll.runtime_pollWait(0xffffb4436198, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0898, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0880, 0x4000110000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0880, 0x4000110000, 0x1000, 0x1000, 0x1311860, 0x47b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000105b0, 0x4000110000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x400010e060, 0x4000111001, 0xdff, 0xdff, 0x400009d8e8, 0x6902d8, 0x40000f5948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x4000112000, 0x4000111001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x4000132000, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4000132000, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4000132000, 0xf5a960, 0x400010e140, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x400010e120, 0x40001340e0, 0x40000f5b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x4000124000, 0x12f8f98, 0x400010e120, 0x4000116867, 0x16, 0x400013a120, 0x400009dc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x4000124000, 0x12f8f98, 0x400010e120, 0x4000124000, 0x4000116070, 0x4000116080, 0x4000130000, 0x4000134000, 0xee0980, 0x400013a120, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x4000124000, 0x12f8f98, 0x400010e120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x40000fa3c0, 0x400009df78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x4000099f78, 0x40000fa3c0, 0x12db8a8, 0x400000e6c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x40000fa3c0, 0x40000b1400, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x40000fa3c0, 0x40000b1400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 10 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 19 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400010c100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 11 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 12 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 13 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 14 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 15 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40000547a0, 0x4, 0x4000054668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x40000ae000, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 53 [IO wait]: internal/poll.runtime_pollWait(0xffffb44360b0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0080, 0x4000430000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0080, 0x4000430000, 0x1000, 0x1000, 0x4000065a88, 0x68f7a4, 0x40000ce470) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010008, 0x4000430000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400031c000, 0x4000430000, 0x1000, 0x1000, 0xf12180, 0x400008a130, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400032f3e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400032f3e0, 0xa, 0x1008880, 0x400008a0f0, 0x199, 0x1000, 0x4000065b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400032f3e0, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000372750) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000372750, 0x40023d4000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40003023c0, 0x40023d4000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40003023c0, 0x40023d4000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000302400, 0x40023d4000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40000ce420, 0x0, 0x4000065eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40000ce420, 0x0, 0x0, 0x4000065f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40000ce420, 0xee0d40, 0x400008a0f0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400147c1b0, 0x400032f740, 0x4000038720) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 51 [select]: net/http.(*persistConn).readLoop(0x400031c000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 52 [select]: net/http.(*persistConn).writeLoop(0x400031c000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 66 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000194450, 0x12c18bc, 0x1, 0x400009fe58, 0x40014366c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x40014256c8, 0x40014366c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400009ffa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4001444000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 67 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4001444000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 76 [select]: net/http.(*persistConn).readLoop(0x400031c240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 70 [IO wait]: internal/poll.runtime_pollWait(0xffffb4435fc8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40023ce118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40023ce100, 0x40003a4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40023ce100, 0x40003a4000, 0x1000, 0x1000, 0x400038abe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400143e060, 0x40003a4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40023ca480, 0x40003a4000, 0x1000, 0x1000, 0x400038acf8, 0x67867c, 0x40003a2360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4001434480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4001434480, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40023ca480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 71 [select]: net/http.(*persistConn).writeLoop(0x40023ca480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 24 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4001444440) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 61 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000194450, 0x1144559, 0x1b, 0x4001451e38, 0x40045bb0e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004581860, 0x40045bb0e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x4002467110) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4001451f80, 0x0, 0x1000000000001) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004586c60, 0x4004581860) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 60 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x400440c960, 0x40045bafc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 72 [IO wait]: internal/poll.runtime_pollWait(0xffffb4435970, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40023ce598, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40023ce580, 0x4000389000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40023ce580, 0x4000389000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400143e098, 0x4000389000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400031c240, 0x4000389000, 0x1000, 0x1000, 0x4000102900, 0x200000003, 0x4000102900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000384960) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000384960, 0x40023de00a, 0x400009d8c8, 0x4d3b10, 0x40000ae060, 0x40023de000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000384960, 0x0, 0x1, 0x0, 0x40000ae078, 0x40023de000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4001438e40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4001438e40, 0x40044a2000, 0x1000, 0x1000, 0x400009d9b8, 0x0, 0x400009d988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000302800, 0x40044a2000, 0x1000, 0x1000, 0x4000102480, 0x3, 0x400009dbe0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000302800, 0x40044a2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000302880, 0x40044a2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400009ddd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400009ddd0, 0xa, 0x40044a2201, 0x1, 0xd20, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400009ddd0, 0x40044a220a, 0x1, 0x4000478fe9, 0x0, 0x40044a22e0, 0x1, 0xd20, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400009ddd0, 0x40000e560a, 0x4000478fe9, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4001442d40, 0x400003a648, 0x17, 0x400009df20, 0x400036a840, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x4001442d40, 0x113075a, 0x7, 0x400036a840, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x4001442d00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 84 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000126f80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 78 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000126340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 28 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x400011c540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 42 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40014458c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 38 [select]: net/http.(*persistConn).readLoop(0x400013b680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 31 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x400011c540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 32 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x400011c540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 33 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x400011c540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 73 [IO wait]: internal/poll.runtime_pollWait(0xffffb4435888, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000131018, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000131000, 0x400441a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000131000, 0x400441a000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40023fe008, 0x400441a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400013b680, 0x400441a000, 0x1000, 0x1000, 0x4000103680, 0x200000003, 0x4000103680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40000ae7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000ae7e0, 0x40023de00a, 0x400009b8c8, 0x4d3b10, 0x40000ae060, 0x40023de000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40000ae7e0, 0x0, 0x1, 0x0, 0x40000ae078, 0x40023de000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40003737a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40003737a0, 0x40003a9000, 0x1000, 0x1000, 0x400009b9b8, 0x44d8a8, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40014446c0, 0x40003a9000, 0x1000, 0x1000, 0x40003846b8, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40014446c0, 0x40003a9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4001444700, 0x40003a9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400009bdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400009bdd0, 0xa, 0x40003a9201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400009bdd0, 0x40003a920a, 0x1, 0x400047927b, 0x0, 0x40003a9263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400009bdd0, 0x40023cc40a, 0x400047927b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4001442d40, 0x4001474510, 0x24, 0x400009bf20, 0x400036a828, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x4001442d40, 0x113075a, 0x7, 0x400036a828, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x4001442d00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 39 [select]: net/http.(*persistConn).writeLoop(0x400013b680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 77 [select]: net/http.(*persistConn).writeLoop(0x400031c240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 85 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x400032fc80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 86 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x4001439080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 87 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x4000130680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 88 [IO wait]: internal/poll.runtime_pollWait(0xffffb44357a0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000130818, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000130800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000130800, 0x4000382000, 0x4000052e98, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004492948, 0x1, 0x4000052f00, 0x46e7e8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004492948, 0x12ed9d0, 0x4004492948, 0x400004f800, 0x4000052f28) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x4000130700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 89 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40000e55f0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 90 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40000e55f0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 91 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x40000e55f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 92 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x400011a9c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 98 [IO wait]: internal/poll.runtime_pollWait(0xffffb44354e8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000131498, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000131480, 0x4000427000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000131480, 0x4000427000, 0x1000, 0x1000, 0x4000439688, 0x69f178, 0x40004396f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000106f0, 0x4000427000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40023cad80, 0x4000427000, 0x1000, 0x1000, 0x5, 0x4004444500, 0x40004397c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000422de0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000422de0, 0x68f90a, 0x40001422c0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000422de0, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4002467080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4002467080, 0x4001456000, 0x1000, 0x1000, 0x4000439998, 0x44e884, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000081d00, 0x4001456000, 0x1000, 0x1000, 0x40003846b8, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000081d00, 0x4001456000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000081d40, 0x4001456000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000439d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000439d80, 0xa, 0x4001456201, 0x1, 0xffff8c5b0648, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000439d80, 0x400145620a, 0x1, 0x40014763b0, 0x0, 0x4001456259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000439d80, 0x400246460a, 0x40014763b0, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4001442d40, 0x4004494560, 0x1f, 0x4000439ee0, 0x4004492c78, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x4001442d40, 0x113075a, 0x7, 0x4000479000, 0xd, 0x4004492c78, 0x0, 0x4000380050, 0x4000384000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x4001442d00, 0x4000479000, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 96 [IO wait]: internal/poll.runtime_pollWait(0xffffb44356b8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000131198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000131180, 0x4000424000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000131180, 0x4000424000, 0x1000, 0x1000, 0x400017abe0, 0x469b10, 0x400441c300) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010688, 0x4000424000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40023ca6c0, 0x4000424000, 0x1000, 0x1000, 0x400017acf8, 0x67867c, 0x40000392c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40004225a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40004225a0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40023ca6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 95 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40000e01c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 97 [select]: net/http.(*persistConn).writeLoop(0x40023ca6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 147 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x400001ec40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 105 [select]: net/http.(*persistConn).readLoop(0x40023cab40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 106 [select]: net/http.(*persistConn).writeLoop(0x40023cab40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 103 [select]: net/http.(*persistConn).readLoop(0x40023cad80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 104 [select]: net/http.(*persistConn).writeLoop(0x40023cad80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 62 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 79 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001271c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 80 [IO wait]: internal/poll.runtime_pollWait(0xffffb4435a58, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40023cf418, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40023cf400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40023cf400, 0x58ec48, 0x4002562960, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004649b18, 0x4714f4, 0x4002562960, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004649b18, 0x4002551800, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x4004670140) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 81 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x4004670580, 0x400007f420, 0x400007f490, 0x400457f630, 0x12e8f90, 0x4004458460, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x4004601914, 0x5, 0x4004670140, 0x12eca70, 0x4004670000, 0x130d478, 0x40003007e0, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x400001ea80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x400001ea80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 114 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x4004458460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 115 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x4004458460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 116 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x4002476b40, 0x400453d500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 117 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x4000126300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 118 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x400440c960) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 119 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x400440c960) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 120 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x400143e580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 121 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x400440ca00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 122 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x400440ca00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 123 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x4004670580, 0x400457f630) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 63 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x4000380b90, 0xd00007, 0x40045e8d38, 0x5, 0x4004581870, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 64 [IO wait]: internal/poll.runtime_pollWait(0xffffb4435400, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1a98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f1a80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f1a80, 0xffffb44df108, 0x30, 0x40045858f0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40045e1998, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40045e1998, 0x40045858f0, 0x11ae500, 0x400011ce80, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x400011ce00, 0x12ed9d0, 0x40045e1998, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x400011ce00, 0x12ed9d0, 0x40045e1998, 0x400143a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 65 [IO wait]: internal/poll.runtime_pollWait(0xffffb4435318, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1b98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f1b80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f1b80, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40045e19b0, 0xffffb44df108, 0xffff8c0f93a0, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40045e19b0, 0x665e1c, 0x0, 0xfe5e60, 0x4000331dc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x40045e19c8, 0x4004585980, 0x11ae500, 0x400011cf60, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x400011cee0, 0x12ec710, 0x40045e19c8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x400011cee0, 0x12ec710, 0x40045e19c8, 0x400143a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 43 [IO wait]: internal/poll.runtime_pollWait(0xffffb44355d0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000131398, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000131380, 0x4000457000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000131380, 0x4000457000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010700, 0x4000457000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40023cab40, 0x4000457000, 0x1000, 0x1000, 0x20, 0x30, 0x20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000422f60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000422f60, 0xa, 0x0, 0x0, 0x40014273e8, 0x597c30, 0x4001427438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000422f60, 0x41ab8c, 0xffff8c1db700, 0x100100000000, 0xffff8c37cfff, 0x400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40047352f0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40047352f0, 0x4004b9ae00, 0x200, 0x200, 0x4001427508, 0x41cff4, 0xffff8c1db700) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000127b40, 0x4004b9ae00, 0x200, 0x200, 0xffff8c15d940, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000127b40, 0x4004b9ae00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000127b80, 0x4004b9ae00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004bce000, 0x0, 0x40014276b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004bce000, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004bce000, 0xee0d40, 0x4000124a50, 0x12db068, 0x4000127b80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004bca000, 0x4004731020, 0x40046f5c20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 45 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4004670a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 146 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x400001ec40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 131 [IO wait]: internal/poll.runtime_pollWait(0xffff8c5d12c0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1e18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f1e00, 0x4004d08000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f1e00, 0x4004d08000, 0x1000, 0x1000, 0x400045bbe0, 0x469b10, 0x400441d800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40023fe250, 0x4004d08000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004676c60, 0x4004d08000, 0x1000, 0x1000, 0x400045bcf8, 0x67867c, 0x40045bb440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004587140) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004587140, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004676c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 132 [select]: net/http.(*persistConn).writeLoop(0x4004676c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 46 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x40003fb8c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 129 [semacquire]: sync.runtime_Semacquire(0x4004e44088) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004e44080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x400469b600, 0x40014388d0, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffd4eb5969, 0x29, 0xffffd4eb59cb, 0x25, 0x4004bfcc00, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004678600, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffd4eb5969, 0x29, 0xffffd4eb59cb, 0x25, 0x4004bfcc00, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004678600, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004419190, 0x4004418e60, 0x40014388d0, 0x400036cb40, 0x4001444440, 0x0, 0x1bf, 0x400144eda8, 0xffff8c1603c0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x400143a000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004678600, 0xb, 0x40046785f8, 0x6, 0x4004678610, 0xd, 0x400464b760, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x400143a000, 0x4000320000, 0x4001438030) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x400143a000, 0x4000012760, 0x2, 0x2, 0x4000012770, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 44 [IO wait]: internal/poll.runtime_pollWait(0xffff8c5d11d8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40023cf998, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40023cf980, 0x4004cfd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40023cf980, 0x4004cfd000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400143e6f0, 0x4004cfd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004676ea0, 0x4004cfd000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004655ce0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004655ce0, 0xa, 0x0, 0x0, 0x40014273e8, 0x597c30, 0x4001427438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004655ce0, 0x41ab8c, 0xffff8c1db700, 0x100100000000, 0xffff8c37cfff, 0x400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004735c80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004735c80, 0x4004b9bc00, 0x200, 0x200, 0x4001427508, 0x41cff4, 0xffff8c1db700) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004e06000, 0x4004b9bc00, 0x200, 0x200, 0xffff8c15d940, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004e06000, 0x4004b9bc00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004e06040, 0x4004b9bc00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004bcfb80, 0x0, 0x40014276b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004bcfb80, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004bcfb80, 0xee0d40, 0x4000124be0, 0x12db068, 0x4004e06040) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004bca990, 0x4004731560, 0x4004bfc600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 148 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x400001ec40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 149 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x400001ec40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 150 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4004670a40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 108 [IO wait]: internal/poll.runtime_pollWait(0xffff8c5d13a8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40023cf898, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40023cf880, 0x4000477000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40023cf880, 0x4000477000, 0x1000, 0x1000, 0x400045ebe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010740, 0x4000477000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004676d80, 0x4000477000, 0x1000, 0x1000, 0x400045ecf8, 0x67867c, 0x4000039c80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40004231a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40004231a0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004676d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 153 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40003fba20, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 154 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40003fba20, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 155 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x40003fba20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 156 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x400001ec40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 157 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000194450, 0x1144559, 0x1b, 0x4000311e38, 0x4004dd6120, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x400457fbf0, 0x4004dd6120, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000311f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004655080, 0x400457fbf0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 160 [select]: net/http.(*persistConn).readLoop(0x4004676ea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 161 [select]: net/http.(*persistConn).writeLoop(0x4004676ea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 109 [select]: net/http.(*persistConn).writeLoop(0x4004676d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 47 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x40003fb8c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 48 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x40003fb8c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 49 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x40003fb8c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 162 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004bbd800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 163 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x4004bbd800, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 164 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x400143a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 165 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x400143a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 166 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x400143a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 167 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:49:57.928+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:49:57.929+00:00 [Info] Setting ipv6=false 2021-08-16T13:49:57.944+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:49:57.951+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:49:57.951+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:49:57.951+00:00 [Info] Setting log level to Info 2021-08-16T13:49:57.952+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:49:57.955+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:49:57 audit: created new audit service 2021-08-16T13:49:57.963+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:49:57.963+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:49:57.964+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:49:57.964+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:49:57.964+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:49:57.975+00:00 [Info] security context initialized 2021-08-16T13:49:57.975+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:49:57.975+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:49:57.975+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:49:57.975+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:49:57.975+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:49:57.988+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:49:57.989+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:49:57.992+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:49:58.010+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:49:58.011+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:49:58.012+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:49:58.014+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:49:58.014+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:49:58.016+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:49:58.016+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:49:58.016+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:49:58.062+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:49:58.062+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 53 us 2021-08-16T13:49:58.062+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:49:58.064+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:49:58.064+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:49:58.068+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:49:58.068+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:49:58.068+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:49:58.068+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:49:58.068+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:49:58.068+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:58.069+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:58.070+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:49:58.070+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:49:58.070+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:49:58.071+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:58.071+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:49:58.071+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:49:58.071+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:49:58.071+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:49:58.071+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:49:58.071+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:49:58.071+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:49:58.078+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 6.532637ms 2021-08-16T13:49:58.078+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:49:58.079+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:49:58.079+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:49:58.079+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:49:58.079+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:49:58.079+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:58.079+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:49:58.090+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:49:58.090+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:49:58.090+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:49:58.091+00:00 [Info] RebalanceMgr::initService Init 2021-08-16T13:49:58.091+00:00 [Info] ServiceMgr::registerWithServer nodeuuid 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:49:58.091+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards unexpected fault address 0x4004c6d45f fatal error: fault 2021-08-16T13:49:58.091+00:00 [Info] : Recover shard shard1 from metadata completed. [signal SIGBUS: bus error code=0x1 addr=0x4004c6d45f pc=0xbd8ea4] goroutine 187 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004d549f0 sp=0x4004d549c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004d54a30 sp=0x4004d549f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x4004c1f040, 0x4004e96600, 0x11a9890, 0x4004d54b40, 0x4004c1f058, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004d54ab0 sp=0x4004d54a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x4004c1f040, 0x4004e96600, 0x11a9890, 0x4004d54b40, 0x4004c1f058, 0xc18f8c, 0x4004d54c18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004d54af0 sp=0x4004d54ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x400447a8c0, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004d54da0 sp=0x4004d54af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x400447a8c0, 0x400235b6c0, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004d54e60 sp=0x4004d54da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004e8a6a0, 0x4004e8a6b0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004d55200 sp=0x4004d54e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004e8a6a0, 0x4004e8a6b0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004d558f0 sp=0x4004d55200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004ed4550, 0x4004d17c00, 0x400467d080, 0x4004e8a680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004d55fb0 sp=0x4004d558f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004d55fb0 sp=0x4004d55fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x40000e2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x40000fb440, 0x4, 0x1136a87, 0xf, 0x40000f9e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 7 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400000e690) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 8 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400000e6a8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 9 [IO wait]: internal/poll.runtime_pollWait(0xffff885d7848, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0e18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0e00, 0x40013b9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0e00, 0x40013b9000, 0x1000, 0x1000, 0x1311860, 0x47b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010668, 0x40013b9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x40013ca000, 0x40013de001, 0xdff, 0xdff, 0x400009f8e8, 0x6902d8, 0x40000f5948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x40013c6018, 0x40013de001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40013b4160, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40013b4160, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40013b4160, 0xf5a960, 0x40001200e0, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x40001200c0, 0x40013d80e0, 0x40000f5b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x40013d4000, 0x12f8f98, 0x40001200c0, 0x40013b6887, 0x16, 0x40013ac360, 0x400009fc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x40013d4000, 0x12f8f98, 0x40001200c0, 0x40013d4000, 0x40013b6090, 0x40013b60a0, 0x40013d6000, 0x40013d8000, 0xee0980, 0x40013ac360, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x40013d4000, 0x12f8f98, 0x40001200c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x40000fa3c0, 0x400009ff78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400009bf78, 0x40000fa3c0, 0x12db8a8, 0x400000ed20, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x40000fa3c0, 0x40000b1360, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x40000fa3c0, 0x40000b1360) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 10 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 11 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 12 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 13 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 14 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 15 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40000547a0, 0x4, 0x4000054668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 18 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400000ac00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 37 [select]: net/http.(*persistConn).writeLoop(0x400035a900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 68 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001943c0, 0x12c18bc, 0x1, 0x400009de58, 0x4000038ae0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x400001c6c8, 0x4000038ae0, 0x400001c6c8, 0x679704) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400009dfa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4000080000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 66 [select]: net/http.(*persistConn).readLoop(0x400000c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 67 [select]: net/http.(*persistConn).writeLoop(0x400000c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 69 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4000080000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 38 [IO wait]: internal/poll.runtime_pollWait(0xffff885d7760, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0180, 0x4000016000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0180, 0x4000016000, 0x1000, 0x1000, 0x4000066a88, 0x68f7a4, 0x40013b4470) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000014000, 0x4000016000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c5a0, 0x4000016000, 0x1000, 0x1000, 0xf12180, 0x40013d40e0, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000082720) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000082720, 0xa, 0x1008880, 0x40013d40a0, 0x199, 0xf03, 0x4000066b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000082720, 0x2, 0x0, 0x9, 0x9b5, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40013c8930) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40013c8930, 0x40013ef000, 0xe00, 0xe00, 0x4000066cc8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40013ba1c0, 0x40013ef000, 0xe00, 0xe00, 0x4000066e08, 0x40bb94, 0x4001394000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40013ba1c0, 0x40013ef000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40013ba200, 0x40013ef000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40013b4420, 0x0, 0x4000066eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40013b4420, 0x0, 0x0, 0x4000066f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40013b4420, 0xee0d40, 0x40013d40a0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40013cc900, 0x4000120360, 0x40013f0120) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 36 [IO wait]: internal/poll.runtime_pollWait(0xffff885d7678, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000340098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000340080, 0x40013df000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000340080, 0x40013df000, 0x1000, 0x1000, 0x4000064be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40013bc168, 0x40013df000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400035a900, 0x40013df000, 0x1000, 0x1000, 0x4000064cf8, 0x67867c, 0x40013f00c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001201e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40001201e0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400035a900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 194 [IO wait]: internal/poll.runtime_pollWait(0xffff884431e0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000340a18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000340a00, 0x4004740000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000340a00, 0x4004740000, 0x1000, 0x1000, 0x0, 0x4004707000, 0x4004d4ba38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000141e8, 0x4004740000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).fill(0x400473e2a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400473e2a0, 0x40a, 0x4004740000, 0x0, 0xffff00001001, 0x400, 0x400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).ReadLine(0x400473e2a0, 0x1000, 0xffff88397370, 0x1, 0x4004d4bbc8, 0x412ff8, 0x400472c510) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:389 +0x30 net/textproto.(*Reader).readLineSlice(0x4004d4bcf0, 0x400472c510, 0x4004d4bc48, 0xffff88397370, 0x2c8, 0xffffb22aa130) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/textproto/reader.go:57 +0xac net/textproto.(*Reader).ReadLine(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/textproto/reader.go:38 net/http.ReadResponse(0x400473e2a0, 0x4004505800, 0x1000, 0x4004740000, 0x40000141e8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/response.go:161 +0x8c github.com/couchbase/cbauth/revrpc.(*Service).Run(0x400472a360, 0x4004d4bf10, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:132 +0x2ac github.com/couchbase/cbauth/revrpc.BabysitService(0x4004d4bf10, 0x400472a360, 0x0, 0x0, 0x0, 0xd8d03c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth/service.RegisterManager(0x13007d8, 0x4004d76800, 0x0, 0x0, 0x4004d5ff80, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:146 +0xa0 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).registerWithServer(0x4004d76800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:301 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:205 +0x78 goroutine 102 [select]: net/http.(*persistConn).readLoop(0x400000c7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 100 [IO wait]: internal/poll.runtime_pollWait(0xffff885d6e50, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004426318, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004426300, 0x400249e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004426300, 0x400249e000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010438, 0x400249e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400253e120, 0x400249e000, 0x1000, 0x1000, 0x4000102c00, 0x200000003, 0x4000102c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002418e40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002418e40, 0x40013f800a, 0x40024fb8c8, 0x4d3b10, 0x40000820c0, 0x40013f8000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4002418e40, 0x0, 0x1, 0x0, 0x40000820d8, 0x40013f8000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40023d5c80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40023d5c80, 0x400450a000, 0x1000, 0x1000, 0x40024fb9b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40013ba740, 0x400450a000, 0x1000, 0x1000, 0x4002518b38, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40013ba740, 0x400450a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40013ba780, 0x400450a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40024fbdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40024fbdd0, 0xa, 0x400450a201, 0x1, 0xd20, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40024fbdd0, 0x400450a20a, 0x1, 0x40045009f9, 0x0, 0x400450a2e0, 0x1, 0xd20, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40024fbdd0, 0x40023ce60a, 0x40045009f9, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40023ce1e0, 0x40013d2b70, 0x17, 0x40024fbf20, 0x40024edd58, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x40023ce1e0, 0x113075a, 0x7, 0x40024edd58, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x40023ce1a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 39 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40013baa40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 104 [IO wait]: internal/poll.runtime_pollWait(0xffff885d7108, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0280, 0x4004508000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0280, 0x4004508000, 0x1000, 0x1000, 0x40003db688, 0x69f178, 0x40003db6f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40013bc1c8, 0x4004508000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c7e0, 0x4004508000, 0x1000, 0x1000, 0x5, 0x400239e000, 0x40003db7c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002518cc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002518cc0, 0x68f90a, 0x4004408e70, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4002518cc0, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400451a570) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400451a570, 0x400452e000, 0x1000, 0x1000, 0x40003db998, 0x44e884, 0x40003db948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40013bab40, 0x400452e000, 0x1000, 0x1000, 0x4002518b38, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40013bab40, 0x400452e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40013bab80, 0x400452e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40003dbd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003dbd80, 0xa, 0x400452e201, 0x1, 0xffff8838d970, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40003dbd80, 0x400452e20a, 0x1, 0x400461c3d0, 0x0, 0x400452e259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40003dbd80, 0x40013b5e0a, 0x400461c3d0, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40023ce1e0, 0x40024eedc0, 0x1f, 0x40003dbee0, 0x4000314090, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x40023ce1e0, 0x113075a, 0x7, 0x4004500a10, 0xd, 0x4000314090, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x40023ce1a0, 0x4004500a10, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 57 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x400241a380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 168 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x400441dc80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 167 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40044789a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 41 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x400241a380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 44 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40024a0040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 58 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x400241a380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 59 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x400241a380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 101 [IO wait]: internal/poll.runtime_pollWait(0xffff885d6d68, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000340318, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000340300, 0x40003c5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000340300, 0x40003c5000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000014060, 0x40003c5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400035ad80, 0x40003c5000, 0x1000, 0x1000, 0x4001397080, 0x200000003, 0x4001397080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40013cac00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40013cac00, 0x40013f800a, 0x40024f98c8, 0x4d3b10, 0x40000820c0, 0x40013f8000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40013cac00, 0x0, 0x1, 0x0, 0x40000820d8, 0x40013f8000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000012e40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000012e40, 0x40003e0000, 0x1000, 0x1000, 0x40024f99b8, 0x0, 0x40024f9988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40001fc640, 0x40003e0000, 0x1000, 0x1000, 0x4000363680, 0x3, 0x40024f9be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40001fc640, 0x40003e0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001fc680, 0x40003e0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40024f9dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40024f9dd0, 0xa, 0x40003e0201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40024f9dd0, 0x40003e020a, 0x1, 0x4000037efb, 0x0, 0x40003e0263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40024f9dd0, 0x400033030a, 0x4000037efb, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40023ce1e0, 0x40013d0750, 0x24, 0x40024f9f20, 0x40024edd40, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x40023ce1e0, 0x113075a, 0x7, 0x40024edd40, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x40023ce1a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 166 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40044789a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 165 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40044789a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 45 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x4002418060) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 46 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x400036e4e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 47 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x4004426080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 48 [IO wait]: internal/poll.runtime_pollWait(0xffff885d7020, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004426198, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4004426180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4004426180, 0x0, 0x40013aae98, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40044e6d38, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40044e6d38, 0x12ed9d0, 0x40044e6d38, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x4004426100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 49 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400441aa90, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 82 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400441aa90, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 83 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x400441aa90) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 84 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x400036fb60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 94 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400441c380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 88 [select]: net/http.(*persistConn).readLoop(0x400253e120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 87 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40000e0230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 26 [select]: net/http.(*persistConn).readLoop(0x400035ad80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 27 [select]: net/http.(*persistConn).writeLoop(0x400035ad80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 89 [select]: net/http.(*persistConn).writeLoop(0x400253e120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 103 [select]: net/http.(*persistConn).writeLoop(0x400000c7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 130 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x40024a1d40, 0x40013e9180, 0x40013e91f0, 0x4004643060, 0x12e8f90, 0x400241ab60, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x4004c173b4, 0x5, 0x40000816c0, 0x12eca70, 0x40024a1b80, 0x130d478, 0x40001881c0, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x400241afc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x400241afc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 112 [select]: net/http.(*persistConn).writeLoop(0x4004424a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 33 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).processSchedIndexes(0x40044efc30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:296 +0x44 created by github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:218 +0x3c goroutine 156 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40044789a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 111 [select]: net/http.(*persistConn).readLoop(0x4004424a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 32 [IO wait]: internal/poll.runtime_pollWait(0xffff88443580, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004427f18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004427f00, 0x40047e5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004427f00, 0x40047e5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40013bc640, 0x40047e5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400253fb00, 0x40047e5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40047f0000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40047f0000, 0xa, 0x0, 0x0, 0x0, 0x0, 0x40024a4c38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40047f0000, 0x0, 0x40024a4c78, 0x655784, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400472a150) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400472a150, 0x40003d7a00, 0x200, 0x200, 0x4, 0x0, 0x40024a4d08) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40001fd000, 0x40003d7a00, 0x200, 0x200, 0x4004429f87, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40001fd000, 0x40003d7a00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001fd040, 0x40003d7a00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004730160, 0x0, 0x40024a4eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004730160, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004730160, 0xee0d40, 0x40013d4730, 0x12db068, 0x40001fd040) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400472c090, 0x40013cbce0, 0x400472e360) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 116 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40024a1400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 115 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40024a0a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 113 [IO wait]: internal/poll.runtime_pollWait(0xffff885d69c8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40013d7c98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40013d7c80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40013d7c80, 0x12d9d48, 0x40024a0b40, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40047c14a0, 0x0, 0x40024f2d80, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40047c14a0, 0x40024a3500, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x40000816c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 117 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x40044d3180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 97 [IO wait]: internal/poll.runtime_pollWait(0xffff885d6b98, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004426a18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004426a00, 0x4002476000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004426a00, 0x4002476000, 0x1000, 0x1000, 0x4004520be0, 0x469b10, 0x400014c000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000106e8, 0x4002476000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400253e5a0, 0x4002476000, 0x1000, 0x1000, 0x4004520cf8, 0x67867c, 0x40024f3860) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002419b60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4002419b60, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400253e5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 114 [select]: net/http.(*persistConn).writeLoop(0x400253e5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 118 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x40044d3180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 119 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x40024a1d40, 0x4004643060) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 120 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x400241ab60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 121 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x400241ab60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 122 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x4002498090, 0x4004653080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 123 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x400441c340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 124 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x40044d3220) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 125 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x40044d3220) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 126 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x40000109a8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 127 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x40044d3220, 0x40046530e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 128 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001943c0, 0x1144559, 0x1b, 0x40024fde38, 0x4004653200, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x40046435d0, 0x4004653200, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40024fdf80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x400464ade0, 0x40046435d0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 129 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 146 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400008d400, 0xd00007, 0x4004c7c1a8, 0x5, 0x40046435e0, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 147 [IO wait]: internal/poll.runtime_pollWait(0xffff885d6c80, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004427998, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4004427980, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4004427980, 0x0, 0x4004d21d88, 0x665fe4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c7e138, 0x611a6d06, 0x4004d21db8, 0x445c24) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c7e138, 0x4004d21e20, 0x4004d21e28, 0x18, 0x400014dc80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x400241b0a0, 0x12ed9d0, 0x4004c7e138, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x400241b0a0, 0x12ed9d0, 0x4004c7e138, 0x40000e2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 148 [IO wait]: internal/poll.runtime_pollWait(0xffff885d6f38, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004427a98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4004427a80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4004427a80, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c7e150, 0xffffb22aaa68, 0xffff8843f0a8, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c7e150, 0x665e1c, 0x0, 0xfe5e60, 0x40044f0dc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004c7e168, 0x4004d18480, 0x11ae500, 0x400241b200, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x400241b180, 0x12ec710, 0x4004c7e168, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x400241b180, 0x12ec710, 0x4004c7e168, 0x40000e2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 157 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001943c0, 0x1144559, 0x1b, 0x4004d4fe38, 0x4004dfe0c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004643bd0, 0x4004dfe0c0, 0x12f86a0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004d4ff80, 0x12edba8, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004bf6960, 0x4004643bd0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 164 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40044789a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 75 [IO wait]: internal/poll.runtime_pollWait(0xffff88443668, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1518, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f1500, 0x4004be7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f1500, 0x4004be7000, 0x1000, 0x1000, 0x400030fbe0, 0x469b10, 0x4004d1a300) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40044ce178, 0x4004be7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400253f9e0, 0x4004be7000, 0x1000, 0x1000, 0x400030fcf8, 0x67867c, 0x40045f8d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400441fb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x400441fb00, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400253f9e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 163 [semacquire]: sync.runtime_Semacquire(0x4004ed4558) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004ed4550) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x400467d080, 0x40000122a0, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffd9e8b969, 0x29, 0xffffd9e8b9cb, 0x25, 0x4004d789c0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004783b00, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffd9e8b969, 0x29, 0xffffd9e8b9cb, 0x25, 0x4004d789c0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004783b00, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004d55190, 0x4004d54e60, 0x40000122a0, 0x4000368d80, 0x40013baa40, 0x0, 0x1bf, 0x4004514da8, 0xffff883a69e8, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x40000e2000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004783b00, 0xb, 0x4004783af8, 0x6, 0x4004783b10, 0xd, 0x40047c3040, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x40000e2000, 0x4000320000, 0x4000096d50) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x40000e2000, 0x400235c210, 0x2, 0x2, 0x400235c220, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 76 [select]: net/http.(*persistConn).writeLoop(0x400253f9e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 154 [IO wait]: internal/poll.runtime_pollWait(0xffff885d6ab0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40013d6e98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40013d6e80, 0x4004526000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40013d6e80, 0x4004526000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40013bc228, 0x4004526000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004424a20, 0x4004526000, 0x1000, 0x1000, 0xffff88164e00, 0x100100000000, 0xffff882f9fff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002519680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002519680, 0x40a, 0x400, 0x200, 0xffff88164e00, 0xffff882f9fff, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4002519680, 0x1, 0x40044f4458, 0x41c330, 0xffff8843eb58, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004d18de0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004d18de0, 0x4004d5a000, 0x200, 0x200, 0x40044f4508, 0x412ddc, 0xffff8843eb58) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004d12240, 0x4004d5a000, 0x200, 0x200, 0xffffb22aaa68, 0x41b9b35, 0x4004d5a000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004d12240, 0x4004d5a000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004d12280, 0x4004d5a000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004c6ec60, 0x0, 0x40044f46b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004c6ec60, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004c6ec60, 0xee0d40, 0x400008da40, 0x12db068, 0x4004d12280) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004c54f30, 0x400464b6e0, 0x4004653800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 132 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40047e60b0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 133 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40047e60b0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 134 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x40047e60b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 135 [select]: net/http.(*persistConn).readLoop(0x400253fb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 136 [select]: net/http.(*persistConn).writeLoop(0x400253fb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 174 [select]: net/http.(*persistConn).writeLoop(0x400035bb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 171 [IO wait]: internal/poll.runtime_pollWait(0xffff88443498, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1c18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f1c00, 0x4004e06000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f1c00, 0x4004e06000, 0x1000, 0x1000, 0x4004d1cbe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40044ce268, 0x4004e06000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40013ad7a0, 0x4004e06000, 0x1000, 0x1000, 0x4004d1ccf8, 0x67867c, 0x4004dfe7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004bf6c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004bf6c60, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40013ad7a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 172 [select]: net/http.(*persistConn).writeLoop(0x40013ad7a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 160 [select]: net/http.(*persistConn).roundTrip(0x40013ad7a0, 0x4004e3e0c0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004e46000, 0xf8, 0x10a8660, 0x400009ee18) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004e46000, 0x1b6e900, 0xc03eb93f856e1c9e, 0x1bfa8423e6) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x40045d5f00, 0x12db048, 0x1b6e900, 0xc03eb93f856e1c9e, 0x1bfa8423e6, 0x1be4a80, 0x0, 0x40044ce2b0, 0x4004e009f0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004e009c0, 0x40045d5f00, 0xc03eb93f856e1c9e, 0x1bfa8423e6, 0x1be4a80, 0x40044ce2b0, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004e009c0, 0x40045d5f00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004e02510, 0x1b, 0x400009f580, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004dd1440, 0x112f905, 0x6, 0x12db908, 0x4004414ec0, 0xee40c0, 0x40000f1d98, 0x400009f580, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x40023d6580, 0x46, 0x12db908, 0x4004414ec0, 0x0, 0x0, 0x0, 0x0, 0xf8cee0, 0x40044ce1b0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x40023d6580, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x20, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400009fd40, 0x4, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x40044fefc0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x40044fefc0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400441dcc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1267 +0xf8 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 161 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x40044efc30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:224 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 178 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x40044efc30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:612 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 179 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x40044efc30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:662 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 180 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x40044efc30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:752 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 181 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004d76800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:249 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 175 [select]: net/http.(*Transport).getConn(0x1b6e900, 0x4004e3e340, 0x0, 0x4004e027e0, 0x4, 0x4004e22e40, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1368 +0x418 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004e46500, 0xf8, 0x10a8660, 0x40024f6588) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:579 +0x5c4 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004e46500, 0x1b6e900, 0xc03eb93f857d5127, 0x1bfa935867) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004e46400, 0x12db048, 0x1b6e900, 0xc03eb93f857d5127, 0x1bfa935867, 0x1be4a80, 0x0, 0x40044ce2e0, 0x4004e00ff0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004e00fc0, 0x4004e46400, 0xc03eb93f857d5127, 0x1bfa935867, 0x1be4a80, 0x40044ce2e0, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004e00fc0, 0x4004e46400, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004e027b0, 0x1b, 0x40024f6cf0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004dd1b00, 0x112f905, 0x6, 0x12db908, 0x4004414f60, 0xee40c0, 0x40000f1f18, 0x40024f6cf0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x40023d6700, 0x46, 0x12db908, 0x4004414f60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x40023d6700, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x4004e00f00, 0xffff8838b5d8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004d594b0, 0x1063520, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x40044ff680, 0x40044ff680, 0x40023d6700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.FetchNewClusterInfoCache(0xffffd9e8b94e, 0xe, 0x113075a, 0x7, 0x1130e44, 0x8, 0x4004e4c0c4, 0x2, 0xf01) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:123 +0x134 github.com/couchbase/indexing/secondary/manager.(*IndexManager).FetchNewClusterInfoCache(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:306 github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).getIndexMetadata(0x1be5520, 0x12ec7a0, 0x4004e3e300, 0x40044d0dc0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:1284 +0x74 github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).handleIndexMetadataRequest(0x1be5520, 0x12edbe0, 0x4004478b60, 0x4004e46300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:1271 +0x3e8 net/http.HandlerFunc.ServeHTTP(0x40046434d0, 0x12edbe0, 0x4004478b60, 0x4004e46300) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2069 +0x40 net/http.(*ServeMux).ServeHTTP(0x4004d12000, 0x12edbe0, 0x4004478b60, 0x4004e46300) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2448 +0x190 net/http.serverHandler.ServeHTTP(0x400241b0a0, 0x12edbe0, 0x4004478b60, 0x4004e46300) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2887 +0xbc net/http.(*conn).serve(0x40001999a0, 0x12f86a0, 0x4004e3e180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:1952 +0x71c created by net/http.(*Server).Serve /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:3013 +0x318 goroutine 183 [select]: net/http.(*persistConn).roundTrip(0x400253e5a0, 0x4004e3e100, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004e46200, 0xf8, 0x10a8660, 0x4004512dd8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004e46200, 0x1b6e900, 0xc03eb93f85709f25, 0x1bfa86a66d) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004e46100, 0x12db048, 0x1b6e900, 0xc03eb93f85709f25, 0x1bfa86a66d, 0x1be4a80, 0x0, 0x40044ce2c0, 0x4004e00ba0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004e00b70, 0x4004e46100, 0xc03eb93f85709f25, 0x1bfa86a66d, 0x1be4a80, 0x40044ce2c0, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004e00b70, 0x4004e46100, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004e02660, 0x1b, 0x4004513540, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004dd1710, 0x112f905, 0x6, 0x12db908, 0x4004414f20, 0xee40c0, 0x40000f1e18, 0x4004513540, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x40023d6680, 0x46, 0x12db908, 0x4004414f20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x40023d6680, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x240, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004513d00, 0x4004e00ab0, 0xffff8838b5d8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x40044ff440, 0xd47600, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x40044ff440, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x40000e2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9702 +0x344 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 184 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x40000e2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1090 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 185 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x40000e2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1017 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 186 [sleep]: time.Sleep(0x12a05f200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/plasma.RunMemQuotaTunerIteration(0x4004d20fb0, 0xdf74d8, 0x4000038960) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:211 +0x24c github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:247 +0xc0 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 goroutine 195 [select]: net/http.(*persistConn).roundTrip(0x400035bb00, 0x40001fd080, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004505700, 0xf8, 0x10a8660, 0x4004d5f798) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004505700, 0x1b6e900, 0xc03eba4d856f9462, 0x1176f89cba2) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004505600, 0x12db048, 0x1b6e900, 0xc03eba4d856f9462, 0x1176f89cba2, 0x1be4a80, 0x0, 0x40000141c8, 0x400472a1e0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x400472a1b0, 0x4004505600, 0xc03eba4d856f9462, 0x1176f89cba2, 0x1be4a80, 0x40000141c8, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x400472a1b0, 0x4004505600, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004724b40, 0x1f, 0x4004d5fd10, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/indexer.getWithAuth(0x4004724b40, 0x1f, 0xe, 0x113846e, 0x11) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:3077 +0x68 github.com/couchbase/indexing/secondary/indexer.getGlobalTopology(0x4004c7db70, 0xe, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:2266 +0x78 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).updateNodeList(0x4004d76800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:220 +0x3c created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:206 +0x98 goroutine 173 [IO wait]: internal/poll.runtime_pollWait(0xffff884433b0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000340918, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000340900, 0x4004e4a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000340900, 0x4004e4a000, 0x1000, 0x1000, 0x40044f0be8, 0x7ff1a8, 0x40044f0cb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40044ce2c8, 0x4004e4a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400035bb00, 0x4004e4a000, 0x1000, 0x1000, 0x60, 0x0, 0x4004724b01) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004bf75c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004bf75c0, 0x1, 0x4004dff440, 0x43d584, 0x0, 0xd8c91c, 0x40044f0d68) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400035bb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 176 [runnable]: net/http.(*connReader).backgroundRead(0x4004e00d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:691 created by net/http.(*connReader).startBackgroundRead /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:688 +0xc4 goroutine 177 [runnable]: net/http.(*Transport).dialConnFor(0x1b6e900, 0x4004e3a370) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1439 created by net/http.(*Transport).queueForDial /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1411 +0x3a8 Initializing write barrier = 8000 2021-08-16T13:50:03.252+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:50:03.252+00:00 [Info] Setting ipv6=false 2021-08-16T13:50:03.262+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:50:03.264+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:50:03.264+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:50:03.264+00:00 [Info] Setting log level to Info 2021-08-16T13:50:03.264+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:50:03.266+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:50:03 audit: created new audit service 2021-08-16T13:50:03.274+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:50:03.274+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:50:03.275+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:50:03.275+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:50:03.275+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:50:03.285+00:00 [Info] security context initialized 2021-08-16T13:50:03.285+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:50:03.285+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:50:03.286+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:50:03.286+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:50:03.286+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:50:03.298+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:50:03.298+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:50:03.303+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:50:03.324+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:50:03.325+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:50:03.327+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:50:03.327+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:50:03.330+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:50:03.330+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:50:03.330+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:50:03.330+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:50:03.378+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:50:03.378+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 54 us 2021-08-16T13:50:03.378+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:50:03.380+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:50:03.381+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:50:03.381+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:50:03.381+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:50:03.381+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:50:03.381+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:50:03.381+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:50:03.385+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:03.385+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:03.387+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:50:03.387+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:50:03.387+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:50:03.387+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:03.389+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:50:03.389+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:50:03.389+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:50:03.389+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:50:03.389+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:50:03.389+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:50:03.389+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:50:03.393+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 3.95266ms 2021-08-16T13:50:03.393+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:50:03.395+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:50:03.395+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:03.395+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:50:03.395+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:50:03.395+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:50:03.395+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:50:03.405+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:50:03.405+00:00 [Info] RebalanceMgr::NewRebalanceMgr false unexpected fault address 0x40047e449f fatal error: fault 2021-08-16T13:50:03.405+00:00 [Info] RebalanceMgr::initService Init 2021-08-16T13:50:03.405+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:50:03.405+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:50:03.405+00:00 [Info] ServiceMgr::registerWithServer nodeuuid 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:03.405+00:00 [Info] : Recover shard shard1 from metadata completed. [signal SIGBUS: bus error code=0x1 addr=0x40047e449f pc=0xbd8ea4] goroutine 178 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004dac9f0 sp=0x4004dac9c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004daca30 sp=0x4004dac9f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x40000a3380, 0x4004651800, 0x11a9890, 0x4004dacb40, 0x40000a3398, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004dacab0 sp=0x4004daca40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x40000a3380, 0x4004651800, 0x11a9890, 0x4004dacb40, 0x40000a3398, 0xc18f8c, 0x4004dacc18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004dacaf0 sp=0x4004dacab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x400464a500, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004dacda0 sp=0x4004dacaf0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x400464a500, 0x4000326d20, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004dace60 sp=0x4004dacda0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x40047e6710, 0x40047e6720, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004dad200 sp=0x4004dace60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x40047e6710, 0x40047e6720, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004dad8f0 sp=0x4004dad200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004ec27a0, 0x40047e1180, 0x4004646b00, 0x40047e66f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004dadfb0 sp=0x4004dad8f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004dadfb0 sp=0x4004dadfb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x4001438000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x4000111710, 0x4, 0x1136a87, 0xf, 0x40000f9e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 18 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x40001380d8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 19 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x40001380f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 20 [IO wait]: internal/poll.runtime_pollWait(0xffff6476ae88, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400018e118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400018e100, 0x40001b5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400018e100, 0x40001b5000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000188010, 0x40001b5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x40001861e0, 0x40000fe001, 0xdff, 0xdff, 0x400009b8e8, 0x6902d8, 0x40000f5948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x4000190048, 0x40000fe001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40000ce840, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40000ce840, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40000ce840, 0xf5a960, 0x40000823e0, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x40000823c0, 0x40000462e0, 0x40000f5b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400008c190, 0x12f8f98, 0x40000823c0, 0x4000036c97, 0x16, 0x400000d0e0, 0x400009bc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400008c190, 0x12f8f98, 0x40000823c0, 0x400008c190, 0x40000364a8, 0x40000364b0, 0x40000f0100, 0x4000046200, 0xee0980, 0x400000d0e0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400008c190, 0x12f8f98, 0x40000823c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4000110690, 0x400009bf78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x4001487f78, 0x4000110690, 0x12db8a8, 0x4000190018, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x4000110690, 0x400012e5a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x4000110690, 0x400012e5a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 8 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400000a700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 21 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 22 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 23 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 24 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 25 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 26 [syscall]: syscall.Syscall(0x3f, 0x0, 0x4000055fa0, 0x4, 0x4000055e68, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x4000055fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x4000055fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x4000055fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 40 [IO wait]: internal/poll.runtime_pollWait(0xffff6476ada0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0280, 0x40000ff000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0280, 0x40000ff000, 0x1000, 0x1000, 0x400006aa88, 0x68f7a4, 0x40000ce5d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400143c028, 0x40000ff000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c5a0, 0x40000ff000, 0x1000, 0x1000, 0xf12180, 0x4000126270, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003f70e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003f70e0, 0xa, 0x1008880, 0x4000126230, 0x199, 0x1000, 0x400006ab00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003f70e0, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4001436990) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4001436990, 0x400007e000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4001442140, 0x400007e000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4001442140, 0x400007e000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4001442180, 0x400007e000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40000ce580, 0x0, 0x400006aeb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40000ce580, 0x0, 0x0, 0x400006af70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40000ce580, 0xee0d40, 0x4000126230, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40000de1b0, 0x40003f7440, 0x4001434ba0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 38 [select]: net/http.(*persistConn).readLoop(0x400000c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 39 [select]: net/http.(*persistConn).writeLoop(0x400000c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 28 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000114780, 0x12c18bc, 0x1, 0x4001481e58, 0x40014346c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x40014156c8, 0x40014346c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4001481fa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4001442000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 29 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4001442000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 54 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000081d40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 32 [IO wait]: internal/poll.runtime_pollWait(0xffff6476acb8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400014a098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400014a080, 0x400001c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400014a080, 0x400001c000, 0x1000, 0x1000, 0x400019bbe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000116070, 0x400001c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001767e0, 0x400001c000, 0x1000, 0x1000, 0x400019bcf8, 0x67867c, 0x40001425a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000082960) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000082960, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40001767e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 33 [select]: net/http.(*persistConn).writeLoop(0x40001767e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 114 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400011d140) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 96 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000081bc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 82 [IO wait]: internal/poll.runtime_pollWait(0xffff6476a2c0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004490498, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004490480, 0x40024fb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004490480, 0x40024fb000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001880c0, 0x40024fb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400447c7e0, 0x40024fb000, 0x1000, 0x1000, 0x4000314d80, 0x200000003, 0x4000314d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001d6e40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001d6e40, 0x400019e00a, 0x400037b8c8, 0x4d3b10, 0x40000820c0, 0x400019e000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40001d6e40, 0x0, 0x1, 0x0, 0x40000820d8, 0x400019e000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4001436ea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4001436ea0, 0x4002556000, 0x1000, 0x1000, 0x400037b9b8, 0x0, 0x400037b988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400011cd40, 0x4002556000, 0x1000, 0x1000, 0x4000314600, 0x3, 0x400037bbe0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400011cd40, 0x4002556000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400011cd80, 0x4002556000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400037bdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400037bdd0, 0xa, 0x4002556201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400037bdd0, 0x400255620a, 0x1, 0x40001c3fb9, 0x0, 0x40025562e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400037bdd0, 0x40001b6f0a, 0x40001c3fb9, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000099490, 0x40001ce570, 0x17, 0x400037bf20, 0x4000370780, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x4000099490, 0x113075a, 0x7, 0x4000370780, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x4000099450) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 56 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x40001aa540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 57 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x40001aa540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 58 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x40001aa540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 59 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x40001aa540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 90 [select]: net/http.(*persistConn).readLoop(0x400447c6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 71 [select]: net/http.(*persistConn).roundTrip(0x4000336fc0, 0x4001443800, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x40045d3200, 0xf8, 0x10a8660, 0x4000436e18) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x40045d3200, 0x1b6e900, 0xc03eb940d8319188, 0x1bfa00643d) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x40045d3100, 0x12db048, 0x1b6e900, 0xc03eb940d8319188, 0x1bfa00643d, 0x1be4a80, 0x0, 0x4000010828, 0x40045ed7d0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x40045ed7a0, 0x40045d3100, 0xc03eb940d8319188, 0x1bfa00643d, 0x1be4a80, 0x4000010828, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x40045ed7a0, 0x40045d3100, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004dc0330, 0x1b, 0x4000437580, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004df6240, 0x112f905, 0x6, 0x12db908, 0x4000118e60, 0xee40c0, 0x40000f1218, 0x4000437580, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x400003e600, 0x46, 0x12db908, 0x4000118e60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x400003e600, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x20, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000437d40, 0x4, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x4004bff200, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x4004bff200, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4004d7a200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1267 +0xf8 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 83 [IO wait]: internal/poll.runtime_pollWait(0xffff6476a3a8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004490318, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004490300, 0x400001b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004490300, 0x400001b000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001880c8, 0x400001b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400447c6c0, 0x400001b000, 0x1000, 0x1000, 0x4000315800, 0x200000003, 0x4000315800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001d6f00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001d6f00, 0x400019e00a, 0x40014838c8, 0x4d3b10, 0x40000820c0, 0x400019e000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40001d6f00, 0x0, 0x1, 0x0, 0x40000820d8, 0x400019e000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4001437230) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4001437230, 0x400256e000, 0x1000, 0x1000, 0x40014839b8, 0x44d8a8, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400011d040, 0x400256e000, 0x1000, 0x1000, 0x40001d69b8, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400011d040, 0x400256e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400011d080, 0x400256e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4001483dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4001483dd0, 0xa, 0x400256e201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4001483dd0, 0x400256e20a, 0x1, 0x400018257b, 0x0, 0x400256e263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4001483dd0, 0x400004680a, 0x400018257b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000099490, 0x40001ca8a0, 0x24, 0x4001483f20, 0x4000370750, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x4000099490, 0x113075a, 0x7, 0x4000370750, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x4000099450) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 14 [IO wait]: internal/poll.runtime_pollWait(0xffff6476a748, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400014ad18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400014ad00, 0x400256f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400014ad00, 0x400256f000, 0x1000, 0x1000, 0x40000a1688, 0x69f178, 0x40000a16f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001161d0, 0x400256f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003ecea0, 0x400256f000, 0x1000, 0x1000, 0x5, 0x4002413400, 0x40000a17c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40024b5380) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40024b5380, 0x68f90a, 0x400017b810, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40024b5380, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40014378c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40014378c0, 0x400257f000, 0x1000, 0x1000, 0x40000a1998, 0x44e884, 0x40000a1948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400011d3c0, 0x400257f000, 0x1000, 0x1000, 0x40001d69b8, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400011d3c0, 0x400257f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400011d400, 0x400257f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40000a1d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000a1d80, 0xa, 0x400257f201, 0x1, 0xffff6454a2b0, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40000a1d80, 0x400257f20a, 0x1, 0x40023e18e0, 0x0, 0x400257f259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40000a1d80, 0x400256310a, 0x40023e18e0, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000099490, 0x4002552300, 0x1f, 0x40000a1ee0, 0x40023dddb8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x4000099490, 0x113075a, 0x7, 0x40001c3fd0, 0xd, 0x40023dddb8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x4000099450, 0x40001c3fd0, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 12 [IO wait]: internal/poll.runtime_pollWait(0xffff6476a660, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400018e798, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400018e780, 0x400252e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400018e780, 0x400252e000, 0x1000, 0x1000, 0x4002520be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010738, 0x400252e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000336fc0, 0x400252e000, 0x1000, 0x1000, 0x4002520cf8, 0x67867c, 0x40000398c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001874a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40001874a0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4000336fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 13 [runnable]: net/http.(*persistConn).writeLoop(0x4000336fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 41 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400011d740) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 42 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x40001864e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 43 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x40004408a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 44 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x400018e000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 45 [IO wait]: internal/poll.runtime_pollWait(0xffff6476a578, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400018e218, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400018e200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400018e200, 0x680da0, 0x4000052698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4002490288, 0x4000142c00, 0x4000083600, 0x4000314b08) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4002490288, 0x12ed9d0, 0x4002490288, 0x58ec48, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x400018e180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 46 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4001440680, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 47 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4001440680, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 48 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x4001440680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 49 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x4000441bc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 73 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x4004d5e580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:612 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 97 [IO wait]: internal/poll.runtime_pollWait(0xffff6476a490, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004491398, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4004491380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4004491380, 0x58ec48, 0x4000143c80, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004bd8528, 0x4714f4, 0x4000143c80, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004bd8528, 0x4002529700, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x40001d1640) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 68 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40000e0150) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 130 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x40001d1a80, 0x40001d2d90, 0x40001d2e00, 0x4004bdca40, 0x12e8f90, 0x40001aab60, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x400473ff94, 0x5, 0x40001d1640, 0x12eca70, 0x40001d1500, 0x130d478, 0x40003d4620, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40024188c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40024188c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 91 [select]: net/http.(*persistConn).writeLoop(0x400447c6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 88 [select]: net/http.(*persistConn).readLoop(0x400447c7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 89 [select]: net/http.(*persistConn).writeLoop(0x400447c7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 72 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x4004d5e580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:224 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 98 [select]: net/http.(*persistConn).readLoop(0x40003ecea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 99 [select]: net/http.(*persistConn).writeLoop(0x40003ecea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 102 [select]: net/http.(*persistConn).readLoop(0x40003ecfc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 103 [select]: net/http.(*persistConn).writeLoop(0x40003ecfc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 106 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4001442cc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 153 [runnable]: internal/poll.runtime_pollWait(0xffff6476c198, 0x77, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004c3c718, 0x77, 0x4000036000, 0x1, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitWrite(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:96 internal/poll.(*FD).WaitWrite(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:528 net.(*netFD).connect(0x4004c3c700, 0x12f8630, 0x4000036098, 0x0, 0x0, 0x12db428, 0x4004c39c40, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:141 +0x1e4 net.(*netFD).dial(0x4004c3c700, 0x12f8630, 0x4000036098, 0x12ff2c0, 0x0, 0x12ff2c0, 0x40046d97d0, 0x0, 0xffff645976d8, 0x10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:149 +0xac net.socket(0x12f8630, 0x4000036098, 0x112eab2, 0x3, 0x2, 0x1, 0x0, 0x0, 0x12ff2c0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:70 +0x148 net.internetSocket(0x12f8630, 0x4000036098, 0x112eab2, 0x3, 0x12ff2c0, 0x0, 0x12ff2c0, 0x40046d97d0, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/ipsock_posix.go:141 +0xcc net.(*sysDialer).doDialTCP(0x4004c3c680, 0x12f8630, 0x4000036098, 0x0, 0x40046d97d0, 0x0, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:65 +0x8c net.(*sysDialer).dialTCP(0x4004c3c680, 0x12f8630, 0x4000036098, 0x0, 0x40046d97d0, 0xf89d40, 0x4004c4e80c, 0x1be4a80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:61 +0xb8 net.(*sysDialer).dialSingle(0x4004c3c680, 0x12f8630, 0x4000036098, 0x12e9300, 0x40046d97d0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:580 +0x4d4 net.(*sysDialer).dialSerial(0x4004c3c680, 0x12f8630, 0x4000036098, 0x4004c4c230, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:548 +0xf8 net.(*Dialer).DialContext(0x4004d93cb8, 0x12f8630, 0x4000036098, 0x112eab2, 0x3, 0x4004c4e80c, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:425 +0x550 net.(*Dialer).Dial(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:348 net.Dial(0x112eab2, 0x3, 0x4004c4e80c, 0xe, 0x30, 0x1008940, 0x1, 0x40046d9770) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:319 +0x7c github.com/couchbase/cbauth/revrpc.(*Service).Run(0x40046d9770, 0x4004d93f10, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:116 +0xc0 github.com/couchbase/cbauth/revrpc.BabysitService(0x4004d93f10, 0x40046d9770, 0x0, 0x0, 0x0, 0xd8d03c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth/service.RegisterManager(0x13007d8, 0x4004eac000, 0x0, 0x0, 0x4004c58f80, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:146 +0xa0 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).registerWithServer(0x4004eac000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:301 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:205 +0x78 goroutine 144 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x4002418c40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 111 [IO wait]: internal/poll.runtime_pollWait(0xffff6476a1d8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400014b118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400014b100, 0x4002573000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400014b100, 0x4002573000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000116218, 0x4002573000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003ecfc0, 0x4002573000, 0x1000, 0x1000, 0x20, 0x30, 0x20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40024b57a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40024b57a0, 0x100a, 0x0, 0x0, 0x4000052398, 0x4af370, 0x4000052438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40024b57a0, 0x7356e4, 0xffffdfc9694e, 0xe, 0x40000524a8, 0x40b2f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40046d9050) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40046d9050, 0x4004c3e800, 0x200, 0x200, 0xffffdfc9694e, 0xe, 0x4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40046fe580, 0x4004c3e800, 0x200, 0x200, 0x4000043f57, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40046fe580, 0x4004c3e800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40046fe5c0, 0x4004c3e800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004c42580, 0x680d00, 0x40000526b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004c42580, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004c42580, 0xee0d40, 0x400008d040, 0x12db068, 0x40046fe5c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004c00d80, 0x40046d4d80, 0x4004573bc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 131 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40001aab60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 132 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40001aab60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 133 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x40024b1a70, 0x400247d440) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 134 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x400011d100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 135 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x4004488dc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 136 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x4004488dc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 137 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x4000188430) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 138 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x4004488e60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 139 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x4004488e60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 140 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x40001d1a80, 0x4004bdca40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 115 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x4004488dc0, 0x40045d0780) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 116 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000114780, 0x1144559, 0x1b, 0x4000439e38, 0x40045d08a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x40045e63d0, 0x40045d08a0, 0x40003ecea0, 0x4002560e40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x40014375f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000439f80, 0x12c2e00, 0x1000000000001) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x400459ec00, 0x40045e63d0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 117 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 118 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x4000126be0, 0xd00007, 0x4004593fd0, 0x5, 0x40045e63e0, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 119 [runnable]: internal/poll.runtime_pollWait(0xffff6476a0f0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0d18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f0d00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f0d00, 0xffff8eebda68, 0x30, 0x40045ec450) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40045e8150, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40045e8150, 0x40045ec450, 0x11ae500, 0x400442c780, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x400442c700, 0x12ed9d0, 0x40045e8150, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x400442c700, 0x12ed9d0, 0x40045e8150, 0x4001438000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 120 [IO wait]: internal/poll.runtime_pollWait(0xffff6476a008, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0e18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f0e00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f0e00, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40045e8168, 0xffff8eebda68, 0xffff645868a8, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40045e8168, 0x665e1c, 0x0, 0xfe5e60, 0x40045f5dc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x40045e8180, 0x40045ec4e0, 0x11ae500, 0x400442c860, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x400442c7e0, 0x12ec710, 0x40045e8180, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x400442c7e0, 0x12ec710, 0x40045e8180, 0x4001438000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 145 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x4002418c40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 142 [semacquire]: sync.runtime_Semacquire(0x4004ec27a8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004ec27a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x4004646b00, 0x4000440810, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffdfc96969, 0x29, 0xffffdfc969cb, 0x25, 0x4004617560, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004bfac80, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffdfc96969, 0x29, 0xffffdfc969cb, 0x25, 0x4004617560, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004bfac80, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004dad190, 0x4004dace60, 0x4000440810, 0x4000175440, 0x4000081d40, 0x0, 0x3f, 0x400009cda8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x4001438000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004bfac80, 0xb, 0x4004bfac78, 0x6, 0x4004bfac90, 0xd, 0x4004bd6e60, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x4001438000, 0x40003f0000, 0x4001436030) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x4001438000, 0x4000184260, 0x2, 0x2, 0x4000184270, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 143 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x4002418c40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 146 [select]: net/http.(*persistConn).writeLoop(0x400000d560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 113 [IO wait]: internal/poll.runtime_pollWait(0xffff6476c450, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004c3c418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004c3c400, 0x4004c46000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004c3c400, 0x4004c46000, 0x1000, 0x1000, 0x4000066be0, 0x469b10, 0x40045ead80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000116560, 0x4004c46000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000d560, 0x4004c46000, 0x1000, 0x1000, 0x4000066cf8, 0x67867c, 0x4004573da0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40046d4de0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40046d4de0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000d560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 162 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x4002418c40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 163 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4004d7a1c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 164 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004d5e6e0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 165 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004d5e6e0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 166 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4004d5e6e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 167 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x4002418c40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 168 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000114780, 0x1144559, 0x1b, 0x400043be38, 0x4004d724e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004bdd700, 0x4004d724e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400043bf80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004cfe7e0, 0x4004bdd700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 173 [IO wait]: internal/poll.runtime_pollWait(0xffff6476c368, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004491b18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004491b00, 0x4004d7f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004491b00, 0x4004d7f000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001885c0, 0x4004d7f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004cf46c0, 0x4004d7f000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004cfec00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004cfec00, 0xa, 0x0, 0x0, 0x0, 0x0, 0x40045f2c38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004cfec00, 0x0, 0x40045f2c78, 0x655784, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004d60fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004d60fc0, 0x4004bdfa00, 0x200, 0x200, 0x4, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004d7a300, 0x4004bdfa00, 0x200, 0x200, 0x4004cf0e07, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004d7a300, 0x4004bdfa00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004d7a340, 0x4004bdfa00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004d789a0, 0x0, 0x40045f2eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004d789a0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004d789a0, 0xee0d40, 0x4004486e60, 0x12db068, 0x4004d7a340) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004d5cbd0, 0x4004cfee40, 0x4004d72ea0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 171 [select]: net/http.(*persistConn).readLoop(0x4004cf46c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 172 [select]: net/http.(*persistConn).writeLoop(0x4004cf46c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 74 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x4004d5e580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:662 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 75 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x4004d5e580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:752 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 174 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).processSchedIndexes(0x4004d5e580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:296 +0x44 created by github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:218 +0x3c goroutine 76 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004eac000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:249 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 155 [runnable]: internal/poll.runtime_pollWait(0xffff6476c280, 0x77, 0x4004eb0000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004c3c618, 0x77, 0x12f8600, 0x40046d5800, 0x4004c3c600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitWrite(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:96 internal/poll.(*FD).WaitWrite(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:528 net.(*netFD).connect(0x4004c3c600, 0x12f8668, 0x40046d5800, 0x0, 0x0, 0x12db428, 0x4004c39c00, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:141 +0x1e4 net.(*netFD).dial(0x4004c3c600, 0x12f8668, 0x40046d5800, 0x12ff2c0, 0x0, 0x12ff2c0, 0x40046d96e0, 0x0, 0xffff645976d8, 0x10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:149 +0xac net.socket(0x12f8668, 0x40046d5800, 0x112eab2, 0x3, 0x2, 0x1, 0x0, 0x0, 0x12ff2c0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:70 +0x148 net.internetSocket(0x12f8668, 0x40046d5800, 0x112eab2, 0x3, 0x12ff2c0, 0x0, 0x12ff2c0, 0x40046d96e0, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/ipsock_posix.go:141 +0xcc net.(*sysDialer).doDialTCP(0x4004c3c580, 0x12f8668, 0x40046d5800, 0x0, 0x40046d96e0, 0xf74f40, 0x1c1c218, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:65 +0x8c net.(*sysDialer).dialTCP(0x4004c3c580, 0x12f8668, 0x40046d5800, 0x0, 0x40046d96e0, 0x46e908, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:61 +0xb8 net.(*sysDialer).dialSingle(0x4004c3c580, 0x12f8668, 0x40046d5800, 0x12e9300, 0x40046d96e0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:580 +0x4d4 net.(*sysDialer).dialSerial(0x4004c3c580, 0x12f8668, 0x40046d5800, 0x4004c4c1f0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:548 +0xf8 net.(*Dialer).DialContext(0x40000822a0, 0x12f8668, 0x40046d5800, 0x112eab2, 0x3, 0x4004c34e30, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:425 +0x550 net/http.(*Transport).dial(0x1b6e900, 0x12f8668, 0x40046d5500, 0x112eab2, 0x3, 0x4004c34e30, 0xe, 0x0, 0x4000116560, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1162 +0x178 net/http.(*Transport).dialConn(0x1b6e900, 0x12f8668, 0x40046d5500, 0x0, 0x400456b1a0, 0x4, 0x4004c34e30, 0xe, 0x0, 0x4000176a20, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1600 +0x155c net/http.(*Transport).dialConnFor(0x1b6e900, 0x40023db970) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1442 +0x90 created by net/http.(*Transport).queueForDial /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1411 +0x3a8 goroutine 78 [runnable]: github.com/couchbase/cbauth.SetRequestAuthVia(0x4004c66000, 0x0, 0x0, 0x3, 0x400456b320) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/convenience.go:29 +0xc0 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x400456b2f0, 0x1b, 0x4004d95540, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:537 +0x13c github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004c01440, 0x112f905, 0x6, 0x12db908, 0x4000047800, 0xee40c0, 0x4004c3c798, 0x4004d95540, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x4001472480, 0x46, 0x12db908, 0x4000047800, 0x12f8668, 0x4004531e00, 0x4004c58608, 0x5825f0, 0x2, 0x4004c34e60, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x4001472480, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x240, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004d95d00, 0x40046d9830, 0xffff6455c918) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x40045706c0, 0xd47600, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x40045706c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x4001438000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9702 +0x344 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 79 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x4001438000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1090 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 80 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x4001438000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1017 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 154 [select]: net/http.(*Transport).getConn(0x1b6e900, 0x40046fe8c0, 0x0, 0x400456b1a0, 0x4, 0x4004c34e30, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1368 +0x418 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004531e00, 0xf8, 0x10a8660, 0x4004c58798) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:579 +0x5c4 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004531e00, 0x1b6e900, 0xc03eba4ed8285021, 0x1176efb52d6) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004531d00, 0x12db048, 0x1b6e900, 0xc03eba4ed8285021, 0x1176efb52d6, 0x1be4a80, 0x0, 0x4000116648, 0x40046d95f0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x40046d95c0, 0x4004531d00, 0xc03eba4ed8285021, 0x1176efb52d6, 0x1be4a80, 0x4000116648, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x40046d95c0, 0x4004531d00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004c39be0, 0x1f, 0x4004c58d10, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/indexer.getWithAuth(0x4004c39be0, 0x1f, 0xe, 0x113846e, 0x11) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:3077 +0x68 github.com/couchbase/indexing/secondary/indexer.getGlobalTopology(0x40047bbdc0, 0xe, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:2266 +0x78 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).updateNodeList(0x4004eac000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:220 +0x3c created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:206 +0x98 goroutine 81 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 goroutine 156 [select]: net.(*netFD).connect.func2(0x12f8668, 0x40046d5800, 0x4004c3c600, 0x4004c4e780, 0x4004c4e720) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:118 +0x70 created by net.(*netFD).connect /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:117 +0x1c0 Initializing write barrier = 8000 2021-08-16T13:50:08.564+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:50:08.565+00:00 [Info] Setting ipv6=false 2021-08-16T13:50:08.570+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:50:08.572+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:50:08.572+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:50:08.572+00:00 [Info] Setting log level to Info 2021-08-16T13:50:08.572+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:50:08.575+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:50:08 audit: created new audit service 2021-08-16T13:50:08.582+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:50:08.582+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:50:08.583+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:50:08.583+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:50:08.583+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:50:08.593+00:00 [Info] security context initialized 2021-08-16T13:50:08.593+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:50:08.593+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:50:08.593+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:50:08.593+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:50:08.593+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:50:08.617+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:50:08.617+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:50:08.621+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:50:08.633+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:50:08.634+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:50:08.635+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:50:08.637+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:50:08.650+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:50:08.652+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:50:08.654+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:50:08.654+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:50:08.696+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:50:08.696+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 52 us 2021-08-16T13:50:08.697+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:50:08.698+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:50:08.698+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:50:08.699+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:50:08.699+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:50:08.699+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:50:08.699+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:50:08.699+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:50:08.701+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:08.702+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:08.703+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:50:08.703+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:50:08.703+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:50:08.703+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:08.703+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:50:08.703+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:50:08.703+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:50:08.703+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:50:08.703+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:50:08.703+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:50:08.704+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:50:08.709+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 5.239314ms 2021-08-16T13:50:08.709+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:50:08.710+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:50:08.710+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:08.710+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:50:08.711+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:50:08.711+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:50:08.711+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:50:08.721+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:50:08.721+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:50:08.721+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:50:08.721+00:00 [Info] RebalanceMgr::initService Init 2021-08-16T13:50:08.721+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:50:08.721+00:00 [Info] ServiceMgr::registerWithServer nodeuuid 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:08.721+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x40024e2d9f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x40024e2d9f pc=0xbd8ea4] goroutine 199 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004f869f0 sp=0x4004f869c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004f86a30 sp=0x4004f869f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x40024e1ba0, 0x4004e56600, 0x11a9890, 0x4004f86b40, 0x40024e1bb8, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004f86ab0 sp=0x4004f86a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x40024e1ba0, 0x4004e56600, 0x11a9890, 0x4004f86b40, 0x40024e1bb8, 0xc18f8c, 0x4004f86c18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004f86af0 sp=0x4004f86ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4004600780, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004f86da0 sp=0x4004f86af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4004600780, 0x40024d1030, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004f86e60 sp=0x4004f86da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x40047cd840, 0x40047cd850, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004f87200 sp=0x4004f86e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x40047cd840, 0x40047cd850, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004f878f0 sp=0x4004f87200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004e61610, 0x4004e70700, 0x400467b600, 0x40047cd820) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004f87fb0 sp=0x4004f878f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004f87fb0 sp=0x4004f87fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x40001b8000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x4000204f00, 0x4, 0x1136a87, 0xf, 0x40000f9e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 7 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400000e678) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 8 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400000e690) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 9 [IO wait]: internal/poll.runtime_pollWait(0xffffba209e88, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011c198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011c180, 0x4000139000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011c180, 0x4000139000, 0x1000, 0x1000, 0x1311860, 0x47b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000116018, 0x4000139000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x4000114180, 0x400014a001, 0xdff, 0xdff, 0x400009b8e8, 0x6902d8, 0x40000f5948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400011e048, 0x400014a001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x4000136160, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4000136160, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4000136160, 0xf5a960, 0x4000114260, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4000114240, 0x40001181a0, 0x40000f5b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x4000144000, 0x12f8f98, 0x4000114240, 0x4000110907, 0x16, 0x400012c6c0, 0x400009bc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x4000144000, 0x12f8f98, 0x4000114240, 0x4000144000, 0x4000110110, 0x4000110120, 0x400011c200, 0x40001180c0, 0xee0980, 0x400012c6c0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x4000144000, 0x12f8f98, 0x4000114240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4000097e60, 0x400009bf78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x40000a1f78, 0x4000097e60, 0x12db8a8, 0x400011e018, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x4000097e60, 0x40000b12c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x4000097e60, 0x40000b12c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 10 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 11 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 12 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 13 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 14 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 18 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400019a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 15 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40000547a0, 0x4, 0x4000054668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 23 [IO wait]: internal/poll.runtime_pollWait(0xffffba209da0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011c318, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011c300, 0x400014b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011c300, 0x400014b000, 0x1000, 0x1000, 0x400006aa88, 0x68f7a4, 0x40001369f0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000186038, 0x400014b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400012c120, 0x400014b000, 0x1000, 0x1000, 0xf12180, 0x4000198310, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000180840) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000180840, 0xa, 0x1008880, 0x40001982d0, 0x199, 0x1000, 0x400006ab00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000180840, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000182e10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000182e10, 0x40001fa000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400018c240, 0x40001fa000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400018c240, 0x40001fa000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400018c280, 0x40001fa000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40001369a0, 0x0, 0x400006aeb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40001369a0, 0x0, 0x0, 0x400006af70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40001369a0, 0xee0d40, 0x40001982d0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40001265a0, 0x4000180c00, 0x40001b6c00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 21 [select]: net/http.(*persistConn).readLoop(0x400012c120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 22 [select]: net/http.(*persistConn).writeLoop(0x400012c120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 38 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400021c2c0, 0x12c18bc, 0x1, 0x400017fe58, 0x40001b66c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x4000053ec8, 0x40001b66c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400017ffa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x400018c0c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 39 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x400018c0c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 109 [select]: net/http.(*persistConn).readLoop(0x40024c8c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 42 [IO wait]: internal/poll.runtime_pollWait(0xffffba209cb8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0080, 0x4000012000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0080, 0x4000012000, 0x1000, 0x1000, 0x40001adbe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000116100, 0x4000012000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003da900, 0x4000012000, 0x1000, 0x1000, 0x40001adcf8, 0x67867c, 0x40000387e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001147e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40001147e0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40003da900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 43 [select]: net/http.(*persistConn).writeLoop(0x40003da900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 132 [select]: net/http.(*persistConn).readLoop(0x400012d680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 106 [IO wait]: internal/poll.runtime_pollWait(0xffffba2091d8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011cd98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011cd80, 0x4004582000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011cd80, 0x4004582000, 0x1000, 0x1000, 0x40024df688, 0x69f178, 0x40024df6f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010548, 0x4004582000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40024c8c60, 0x4004582000, 0x1000, 0x1000, 0x5, 0x40024f2000, 0x40024df7c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000115da0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000115da0, 0x68f90a, 0x400439e580, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000115da0, 0x0, 0x40024df868, 0x41a0a8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40023ef530) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40023ef530, 0x40024ef000, 0x1000, 0x1000, 0x40024df998, 0x44e884, 0x1c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40024ce300, 0x40024ef000, 0x1000, 0x1000, 0x4000181318, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40024ce300, 0x40024ef000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40024ce340, 0x40024ef000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40024dfd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40024dfd80, 0xa, 0x40024ef201, 0x1, 0xffff90104c50, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40024dfd80, 0x40024ef20a, 0x1, 0x40047b83c0, 0x0, 0x40024ef259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40024dfd80, 0x400013730a, 0x40047b83c0, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000464e10, 0x400041bb40, 0x1f, 0x40024dfee0, 0x400041cee8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x4000464e10, 0x113075a, 0x7, 0x4000191570, 0xd, 0x400041cee8, 0x0, 0x0, 0x2) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x4000464dd0, 0x4000191570, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 24 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400011ae40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 31 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x400027a7e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 54 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400018ccc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 111 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40003d8240) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 168 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40043e68c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 26 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x400237e380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 66 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x400237e380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 67 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x400237e380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 68 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x400237e380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 133 [select]: net/http.(*persistConn).writeLoop(0x400012d680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 71 [IO wait]: internal/poll.runtime_pollWait(0xffffba2093a8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f1200, 0x4002438000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f1200, 0x4002438000, 0x1000, 0x1000, 0x40001a6be0, 0x469b10, 0x40003e3e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000186100, 0x4002438000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003db0e0, 0x4002438000, 0x1000, 0x1000, 0x40001a6cf8, 0x67867c, 0x4000451aa0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000181da0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000181da0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40003db0e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 77 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400018d180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 55 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x40003a91a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 56 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x4002430b10) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 57 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x40000f0100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 58 [IO wait]: internal/poll.runtime_pollWait(0xffffba209748, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0318, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f0300, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f0300, 0x0, 0x4000462698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4002464528, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4002464528, 0x12ed9d0, 0x4002464528, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x40000f0200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 59 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40000e49c0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 60 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40000e49c0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 61 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x40000e49c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 62 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x40024317a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 87 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x400011ba40, 0x400453ecc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 70 [IO wait]: internal/poll.runtime_pollWait(0xffffba209578, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011c298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011c280, 0x400001a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011c280, 0x400001a000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000100f8, 0x400001a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40024c85a0, 0x400001a000, 0x1000, 0x1000, 0x400019d680, 0x200000003, 0x400019d680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001149c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001149c0, 0x40000f600a, 0x400009b8c8, 0x4d3b10, 0x40000820c0, 0x40000f6000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40001149c0, 0x0, 0x1, 0x0, 0x40000820d8, 0x40000f6000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004394c90) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004394c90, 0x400243a000, 0x1000, 0x1000, 0x400009b9b8, 0x0, 0x400009b988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40003d8740, 0x400243a000, 0x1000, 0x1000, 0x400019c780, 0x3, 0x400009bbe0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40003d8740, 0x400243a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40003d8780, 0x400243a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400009bdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400009bdd0, 0xa, 0x400243a201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400009bdd0, 0x400243a20a, 0x1, 0x4004392eeb, 0x0, 0x400243a263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400009bdd0, 0x400043e50a, 0x4004392eeb, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000464e10, 0x400042ed20, 0x24, 0x400009bf20, 0x400237ca08, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x4000464e10, 0x113075a, 0x7, 0x400237ca08, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x4000464dd0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 69 [IO wait]: internal/poll.runtime_pollWait(0xffffba209490, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011c498, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011c480, 0x40001d4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011c480, 0x40001d4000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010460, 0x40001d4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40024c87e0, 0x40001d4000, 0x1000, 0x1000, 0x400019d800, 0x200000003, 0x400019d800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000115020) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000115020, 0x40000f600a, 0x40024db8c8, 0x4d3b10, 0x40000820c0, 0x40000f6000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000115020, 0x0, 0x1, 0x3f, 0x40000820d8, 0x40000f6000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004394f90) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004394f90, 0x400243f000, 0x1000, 0x1000, 0x40024db9b8, 0x0, 0x40024db988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40003d8880, 0x400243f000, 0x1000, 0x1000, 0x40003e2000, 0x3, 0x40024dbbe0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40003d8880, 0x400243f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40003d88c0, 0x400243f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40024dbdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40024dbdd0, 0xa, 0x400243f201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40024dbdd0, 0x400243f20a, 0x1, 0x4000191559, 0x0, 0x400243f2e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40024dbdd0, 0x40024c710a, 0x4000191559, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000464e10, 0x40004427b0, 0x17, 0x40024dbf20, 0x400237ca20, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x4000464e10, 0x113075a, 0x7, 0x400237ca20, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x4000464dd0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 98 [select]: net/http.(*persistConn).readLoop(0x40024c85a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 99 [select]: net/http.(*persistConn).writeLoop(0x40024c85a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 102 [select]: net/http.(*persistConn).readLoop(0x40024c87e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 103 [select]: net/http.(*persistConn).writeLoop(0x40024c87e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 72 [select]: net/http.(*persistConn).writeLoop(0x40003db0e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 86 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x4004542500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 85 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x4004542500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 110 [select]: net/http.(*persistConn).writeLoop(0x40024c8c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 118 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x400011ba40, 0x40001ed3b0, 0x40001ed420, 0x400453ecc0, 0x12e8f90, 0x40043e6540, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x400472bf84, 0x5, 0x4004cba340, 0x12eca70, 0x4004cba200, 0x130d478, 0x400015e000, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x400237e9a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x400237e9a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 117 [IO wait]: internal/poll.runtime_pollWait(0xffffba2092c0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011d598, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400011d580, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400011d580, 0x58ec48, 0x400458c5a0, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004cb1de8, 0x4714f4, 0x400458c5a0, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004cb1de8, 0x4000191c00, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x4004cba340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 116 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40003d9140) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 119 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40043e6540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 120 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40043e6540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 121 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x400043cf30, 0x4004cb4720) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 122 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x40003d8200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 123 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x4000219900) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 124 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x4000219900) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 125 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x4000010968) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 126 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x4000219900, 0x4004cb4780) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 127 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400021c2c0, 0x1144559, 0x1b, 0x400009fe38, 0x4004cb48a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004cfe150, 0x4004cb48a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400009ff80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004589980, 0x4004cfe150) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 128 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 129 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400008d0e0, 0xd00007, 0x4004cdf7e0, 0x5, 0x4004cfe160, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 146 [runnable]: internal/poll.runtime_pollWait(0xffffba2090f0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011d718, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400011d700, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400011d700, 0xffffba2855b8, 0x30, 0x4004cb7b30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004cf22e8, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004cf22e8, 0x4004cb7b30, 0x11ae500, 0x400237eb00, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x400237ea80, 0x12ed9d0, 0x4004cf22e8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x400237ea80, 0x12ed9d0, 0x4004cf22e8, 0x40001b8000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 147 [IO wait]: internal/poll.runtime_pollWait(0xffffba209008, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011d818, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400011d800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400011d800, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004cf2300, 0xffffba2855b8, 0xffff900a0d00, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004cf2300, 0x665e1c, 0x0, 0xfe5e60, 0x40047e05c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004cf2318, 0x4004cb7bc0, 0x11ae500, 0x400237ebe0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x400237eb60, 0x12ec710, 0x4004cf2318, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x400237eb60, 0x12ec710, 0x4004cf2318, 0x40001b8000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 78 [IO wait]: internal/poll.runtime_pollWait(0xffffba209660, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40024d6d98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40024d6d80, 0x40047d8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40024d6d80, 0x40047d8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400044a208, 0x40047d8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400012d680, 0x40047d8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40047be540) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40047be540, 0x100a, 0x0, 0x0, 0x40001a3be8, 0x597c30, 0x40001a3c38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40047be540, 0x0, 0x40001a3c78, 0x655784, 0x40001a3ca8, 0x40b2f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40048222d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40048222d0, 0x400244f800, 0x200, 0x200, 0x60, 0x0, 0x40001a3d01) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40003d9280, 0x400244f800, 0x200, 0x200, 0x4000451ec0, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40003d9280, 0x400244f800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40003d92c0, 0x400244f800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40048246e0, 0x680d00, 0x40001a3eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40048246e0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40048246e0, 0xee0d40, 0x4000198cd0, 0x12db068, 0x40003d92c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40001273b0, 0x400243d080, 0x40024436e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 169 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40043e68c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 80 [IO wait]: internal/poll.runtime_pollWait(0xffff9038ce80, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000441398, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000441380, 0x400481f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000441380, 0x400481f000, 0x1000, 0x1000, 0x4004831be0, 0x469b10, 0x400019d980) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000186300, 0x400481f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400012d8c0, 0x400481f000, 0x1000, 0x1000, 0x4004831cf8, 0x67867c, 0x4002443920) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400243d380) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x400243d380, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400012d8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 81 [select]: net/http.(*persistConn).writeLoop(0x400012d8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 91 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).processSchedIndexes(0x4004840580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:296 +0x44 created by github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:218 +0x3c goroutine 167 [semacquire]: sync.runtime_Semacquire(0x4004e61618) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004e61610) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x400467b600, 0x400010e5d0, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffc8691969, 0x29, 0xffffc86919cb, 0x25, 0x4004e72120, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004cdec70, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffc8691969, 0x29, 0xffffc86919cb, 0x25, 0x4004e72120, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004cdec70, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004f87190, 0x4004f86e60, 0x400010e5d0, 0x400012f680, 0x400011ae40, 0x0, 0x13f, 0x400009cda8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x40001b8000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004cdec70, 0xb, 0x4004cdec68, 0x6, 0x4004cdec80, 0xd, 0x4004cb2dc0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x40001b8000, 0x40003a0000, 0x40001822d0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x40001b8000, 0x40001127b0, 0x2, 0x2, 0x40001127c0, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 139 [IO wait]: internal/poll.runtime_pollWait(0xffff9038cd98, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000441b18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000441b00, 0x400484f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000441b00, 0x400484f000, 0x1000, 0x1000, 0x4000065ad8, 0x412bc0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001864f0, 0x400484f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004e02120, 0x400484f000, 0x1000, 0x1000, 0x40047bf1a0, 0x60, 0x58) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004e00a80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004e00a80, 0x55b60a, 0xf1f980, 0x0, 0x37, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004e00a80, 0x0, 0x1, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004e3a0c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004e3a0c0, 0x40047e6a00, 0x200, 0x200, 0x2, 0xe, 0x4000175e80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40024cf700, 0x40047e6a00, 0x200, 0x200, 0xffffc86919cb, 0x0, 0x400010e5d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40024cf700, 0x40047e6a00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40024cf740, 0x40047e6a00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40047d5ce0, 0x0, 0x40047dceb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40047d5ce0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40047d5ce0, 0xee0d40, 0x400044f0e0, 0x12db068, 0x40024cf740) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40047af8c0, 0x40047bf3e0, 0x40047bd5c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 170 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40043e68c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 171 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40043e68c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 172 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x40003d9a40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 184 [IO wait]: internal/poll.runtime_pollWait(0xffff9038cf68, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000441a18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000441a00, 0x4004e05000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000441a00, 0x4004e05000, 0x1000, 0x1000, 0x4004e92be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001864f8, 0x4004e05000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004e02000, 0x4004e05000, 0x1000, 0x1000, 0x4004e92cf8, 0x67867c, 0x4004845680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004e00ba0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004e00ba0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004e02000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 175 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40048406e0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 176 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40048406e0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 177 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x40048406e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 178 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40043e68c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 179 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400021c2c0, 0x1144559, 0x1b, 0x40024dde38, 0x4004844960, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004826b70, 0x4004844960, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40024ddf80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x400243dda0, 0x4004826b70) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 185 [select]: net/http.(*persistConn).writeLoop(0x4004e02000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 182 [select]: net/http.(*persistConn).readLoop(0x4004e02120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 183 [select]: net/http.(*persistConn).writeLoop(0x4004e02120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 140 [select]: net/http.(*persistConn).roundTrip(0x4004e02000, 0x4004f5a040, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004553800, 0xf8, 0x10a8660, 0x4004e88e18) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004553800, 0x1b6e900, 0xc03eb9422aff3210, 0x1bfa42c4d6) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004553700, 0x12db048, 0x1b6e900, 0xc03eb9422aff3210, 0x1bfa42c4d6, 0x1be4a80, 0x0, 0x4000116520, 0x4004539c80, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004539c50, 0x4004553700, 0xc03eb9422aff3210, 0x1bfa42c4d6, 0x1be4a80, 0x4000116520, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004539c50, 0x4004553700, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004d7a510, 0x1b, 0x4004e89580, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004f581b0, 0x112f905, 0x6, 0x12db908, 0x4000119260, 0xee40c0, 0x4004f56098, 0x4004e89580, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x40024ac500, 0x46, 0x12db908, 0x4000119260, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x40024ac500, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x20, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004e89d40, 0x4, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x4002450b40, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x4002450b40, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40003d9a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1267 +0xf8 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 141 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x4004840580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:224 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 142 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x4004840580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:612 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 143 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x4004840580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:662 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 144 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x4004840580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:752 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 145 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x40047d7000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:249 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 94 [runnable]: internal/poll.runtime_pollWait(0xffff9038ccb0, 0x77, 0x40047ed080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004f56298, 0x77, 0x12f8600, 0x4004f74060, 0x4004f56280) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitWrite(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:96 internal/poll.(*FD).WaitWrite(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:528 net.(*netFD).connect(0x4004f56280, 0x12f8668, 0x4004f74060, 0x0, 0x0, 0x12db428, 0x4004f53020, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:141 +0x1e4 net.(*netFD).dial(0x4004f56280, 0x12f8668, 0x4004f74060, 0x12ff2c0, 0x0, 0x12ff2c0, 0x4004f76030, 0x0, 0xffff90157aa0, 0x10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:149 +0xac net.socket(0x12f8668, 0x4004f74060, 0x112eab2, 0x3, 0x2, 0x1, 0x0, 0x0, 0x12ff2c0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:70 +0x148 net.internetSocket(0x12f8668, 0x4004f74060, 0x112eab2, 0x3, 0x12ff2c0, 0x0, 0x12ff2c0, 0x4004f76030, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/ipsock_posix.go:141 +0xcc net.(*sysDialer).doDialTCP(0x4004f56200, 0x12f8668, 0x4004f74060, 0x0, 0x4004f76030, 0xf74f40, 0x1c1c218, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:65 +0x8c net.(*sysDialer).dialTCP(0x4004f56200, 0x12f8668, 0x4004f74060, 0x0, 0x4004f76030, 0x46e908, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:61 +0xb8 net.(*sysDialer).dialSingle(0x4004f56200, 0x12f8668, 0x4004f74060, 0x12e9300, 0x4004f76030, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:580 +0x4d4 net.(*sysDialer).dialSerial(0x4004f56200, 0x12f8668, 0x4004f74060, 0x400453f840, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:548 +0xf8 net.(*Dialer).DialContext(0x40000822a0, 0x12f8668, 0x4004f74060, 0x112eab2, 0x3, 0x4004f38950, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:425 +0x550 net/http.(*Transport).dial(0x1b6e900, 0x12f8668, 0x4004501d40, 0x112eab2, 0x3, 0x4004f38950, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1162 +0x178 net/http.(*Transport).dialConn(0x1b6e900, 0x12f8668, 0x4004501d40, 0x0, 0x4004d7a750, 0x4, 0x4004f38950, 0xe, 0x0, 0x4004d4e6c0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1600 +0x155c net/http.(*Transport).dialConnFor(0x1b6e900, 0x4004541290) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1442 +0x90 created by net/http.(*Transport).queueForDial /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1411 +0x3a8 goroutine 195 [select]: net/http.(*persistConn).roundTrip(0x40003db0e0, 0x4004f5a080, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004553a00, 0xf8, 0x10a8660, 0x4004f66dd8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004553a00, 0x1b6e900, 0xc03eb9422b00715b, 0x1bfa440421) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004553900, 0x12db048, 0x1b6e900, 0xc03eb9422b00715b, 0x1bfa440421, 0x1be4a80, 0x0, 0x4000116530, 0x4004539e30, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004539e00, 0x4004553900, 0xc03eb9422b00715b, 0x1bfa440421, 0x1be4a80, 0x4000116530, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004539e00, 0x4004553900, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004d7a630, 0x1b, 0x4004f67540, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004f58480, 0x112f905, 0x6, 0x12db908, 0x40001192c0, 0xee40c0, 0x4004f56198, 0x4004f67540, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x400003e400, 0x46, 0x12db908, 0x40001192c0, 0x400438f598, 0x81ee08, 0x400438f748, 0x95dcbc, 0x40024ac500, 0x46, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x400003e400, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x240, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004f67d00, 0x4004530240, 0xffff900a2460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x4004f5e000, 0xd47600, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x4004f5e000, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x40001b8000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9702 +0x344 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 196 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x40001b8000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1090 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 197 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x40001b8000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1017 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 198 [runnable]: github.com/couchbase/plasma._Cfunc_sigar_pid_get(0x40be34a0, 0x4000000e9d) _cgo_gotypes.go:148 +0x44 github.com/couchbase/plasma.GetProcessRSS.func1.1(0x400019fdd8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:100 +0x54 github.com/couchbase/plasma.GetProcessRSS.func1(0x4004732990, 0x4004732990) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:100 +0x30 github.com/couchbase/plasma.GetProcessRSS(0x1ed374000, 0x1bdf39000, 0x40569ab1ae8dcd0f, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:100 +0x3c github.com/couchbase/plasma.RunMemQuotaTunerIteration(0x400019ffb0, 0xdf74d8, 0x40001b6540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:153 +0xe4 github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:247 +0xc0 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 goroutine 92 [runnable]: internal/poll.runtime_pollWait(0xffff9038cbc8, 0x77, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004f56398, 0x77, 0x4000036000, 0x1, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitWrite(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:96 internal/poll.(*FD).WaitWrite(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:528 net.(*netFD).connect(0x4004f56380, 0x12f8630, 0x4000036098, 0x0, 0x0, 0x12db428, 0x4004f53060, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:141 +0x1e4 net.(*netFD).dial(0x4004f56380, 0x12f8630, 0x4000036098, 0x12ff2c0, 0x0, 0x12ff2c0, 0x4004f76120, 0x0, 0xffff90157aa0, 0x10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:149 +0xac net.socket(0x12f8630, 0x4000036098, 0x112eab2, 0x3, 0x2, 0x1, 0x0, 0x0, 0x12ff2c0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:70 +0x148 net.internetSocket(0x12f8630, 0x4000036098, 0x112eab2, 0x3, 0x12ff2c0, 0x0, 0x12ff2c0, 0x4004f76120, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/ipsock_posix.go:141 +0xcc net.(*sysDialer).doDialTCP(0x4004f56300, 0x12f8630, 0x4000036098, 0x0, 0x4004f76120, 0x0, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:65 +0x8c net.(*sysDialer).dialTCP(0x4004f56300, 0x12f8630, 0x4000036098, 0x0, 0x4004f76120, 0xf89d40, 0x4004d1fb8c, 0x1be4a80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:61 +0xb8 net.(*sysDialer).dialSingle(0x4004f56300, 0x12f8630, 0x4000036098, 0x12e9300, 0x4004f76120, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:580 +0x4d4 net.(*sysDialer).dialSerial(0x4004f56300, 0x12f8630, 0x4000036098, 0x400453f880, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:548 +0xf8 net.(*Dialer).DialContext(0x4004f65cb8, 0x12f8630, 0x4000036098, 0x112eab2, 0x3, 0x4004d1fb8c, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:425 +0x550 net.(*Dialer).Dial(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:348 net.Dial(0x112eab2, 0x3, 0x4004d1fb8c, 0xe, 0x30, 0x1008940, 0x1, 0x4004f760c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:319 +0x7c github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4004f760c0, 0x4004f65f10, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:116 +0xc0 github.com/couchbase/cbauth/revrpc.BabysitService(0x4004f65f10, 0x4004f760c0, 0x0, 0x0, 0x0, 0xd8d03c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth/service.RegisterManager(0x13007d8, 0x40047d7000, 0x0, 0x0, 0x400438ff80, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:146 +0xa0 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).registerWithServer(0x40047d7000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:301 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:205 +0x78 goroutine 93 [select]: net/http.(*Transport).getConn(0x1b6e900, 0x4004f5a0c0, 0x0, 0x4004d7a750, 0x4, 0x4004f38950, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1368 +0x418 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004553c00, 0xf8, 0x10a8660, 0x400438f798) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:579 +0x5c4 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004553c00, 0x1b6e900, 0xc03eba502b01b3f3, 0x1176f4976b9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004553b00, 0x12db048, 0x1b6e900, 0xc03eba502b01b3f3, 0x1176f4976b9, 0x1be4a80, 0x0, 0x4000116540, 0x4004539f20, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004539ef0, 0x4004553b00, 0xc03eba502b01b3f3, 0x1176f4976b9, 0x1be4a80, 0x4000116540, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004539ef0, 0x4004553b00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004f53000, 0x1f, 0x400438fd10, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/indexer.getWithAuth(0x4004f53000, 0x1f, 0xe, 0x113846e, 0x11) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:3077 +0x68 github.com/couchbase/indexing/secondary/indexer.getGlobalTopology(0x4004e60c30, 0xe, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:2266 +0x78 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).updateNodeList(0x40047d7000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:220 +0x3c created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:206 +0x98 goroutine 95 [select]: net.(*netFD).connect.func2(0x12f8668, 0x4004f74060, 0x4004f56280, 0x4004d1fb00, 0x4004d1faa0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:118 +0x70 created by net.(*netFD).connect /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:117 +0x1c0 Initializing write barrier = 8000 2021-08-16T13:50:13.943+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:50:13.943+00:00 [Info] Setting ipv6=false 2021-08-16T13:50:13.949+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:50:13.951+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:50:13.951+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:50:13.951+00:00 [Info] Setting log level to Info 2021-08-16T13:50:13.951+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:50:13.953+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:50:13 audit: created new audit service 2021-08-16T13:50:13.961+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:50:13.961+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:50:13.961+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:50:13.961+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:50:13.961+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:50:13.971+00:00 [Info] security context initialized 2021-08-16T13:50:13.972+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:50:13.972+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:50:13.972+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:50:13.972+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:50:13.972+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:50:13.985+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:50:13.987+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:50:13.987+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:50:14.010+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:50:14.010+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:50:14.012+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:50:14.015+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:50:14.017+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:50:14.018+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:50:14.020+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:50:14.020+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:50:14.062+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:50:14.063+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 72 us 2021-08-16T13:50:14.063+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:50:14.065+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:50:14.065+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:50:14.069+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:50:14.069+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:50:14.069+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:50:14.069+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:50:14.069+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:50:14.069+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:14.069+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:14.074+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:50:14.074+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:50:14.074+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:50:14.074+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:14.074+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:50:14.074+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:50:14.074+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:50:14.074+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:50:14.074+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:50:14.074+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:50:14.074+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:50:14.078+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 4.135789ms 2021-08-16T13:50:14.079+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:50:14.079+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:50:14.079+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:14.080+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:50:14.080+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:50:14.080+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:50:14.080+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:50:14.090+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:50:14.090+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:50:14.090+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:50:14.090+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:50:14.090+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x400017dd5f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x400017dd5f pc=0xbd8ea4] goroutine 189 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004dde9f0 sp=0x4004dde9c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004ddea30 sp=0x4004dde9f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x4004e8e4e0, 0x4004f30000, 0x11a9890, 0x4004ddeb40, 0x4004e8e4f8, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004ddeab0 sp=0x4004ddea40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x4004e8e4e0, 0x4004f30000, 0x11a9890, 0x4004ddeb40, 0x4004e8e4f8, 0xc18f8c, 0x4004ddec18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004ddeaf0 sp=0x4004ddeab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x40001a28c0, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004ddeda0 sp=0x4004ddeaf0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x40001a28c0, 0x4000345420, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004ddee60 sp=0x4004ddeda0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004f0e1f0, 0x4004f0e200, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004ddf200 sp=0x4004ddee60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004f0e1f0, 0x4004f0e200, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004ddf8f0 sp=0x4004ddf200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004e81f80, 0x4004f00e00, 0x40001ce100, 0x4004f0e1d0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004ddffb0 sp=0x4004ddf8f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004ddffb0 sp=0x4004ddffb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x40001c0700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x4000116f90, 0x4, 0x1136a87, 0xf, 0x40000f9e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 7 [IO wait]: internal/poll.runtime_pollWait(0xffffbc228da0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000188198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000188180, 0x40001fb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000188180, 0x40001fb000, 0x1000, 0x1000, 0x4000308a88, 0x68f7a4, 0x40000ce050) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000100d0, 0x40001fb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c5a0, 0x40001fb000, 0x1000, 0x1000, 0xf12180, 0x4000406180, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000082660) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000082660, 0xa, 0x1008880, 0x4000406140, 0x199, 0x1000, 0x4000308b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000082660, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40001163c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40001163c0, 0x400017e000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000124000, 0x400017e000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000124000, 0x400017e000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000124040, 0x400017e000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40000ce000, 0x0, 0x4000308eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40000ce000, 0x0, 0x0, 0x4000308f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40000ce000, 0xee0d40, 0x4000406140, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40000fa000, 0x40003d6720, 0x400014c120) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 18 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x4000180108) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 19 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x4000180120) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 20 [IO wait]: internal/poll.runtime_pollWait(0xffffbc228e88, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000128118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000128100, 0x40001f5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000128100, 0x40001f5000, 0x1000, 0x1000, 0x1311860, 0x47b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000190088, 0x40001f5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x40001b06c0, 0x40001fa001, 0xdff, 0xdff, 0x400009b8e8, 0x6902d8, 0x40000f5948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x4000180150, 0x40001fa001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x400019c2c0, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400019c2c0, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400019c2c0, 0xf5a960, 0x40001b07a0, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x40001b0780, 0x400018a4a0, 0x40000f5b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x4000192690, 0x12f8f98, 0x40001b0780, 0x4000184fa7, 0x16, 0x40001ec240, 0x400009bc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x4000192690, 0x12f8f98, 0x40001b0780, 0x4000192690, 0x40001847b8, 0x40001847c0, 0x4000188700, 0x400018a3c0, 0xee0980, 0x40001ec240, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x4000192690, 0x12f8f98, 0x40001b0780) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4000186390, 0x400009bf78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x40000a1f78, 0x4000186390, 0x12db8a8, 0x400012e000, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x4000186390, 0x40001a0460, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x4000186390, 0x40001a0460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 22 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x40001ac500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 34 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 35 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 36 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 37 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 38 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 39 [syscall]: syscall.Syscall(0x3f, 0x0, 0x400042c7a0, 0x4, 0x400042c668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x400042c7a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x400042c7a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x400042c7a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 46 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000081b80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 44 [select]: net/http.(*persistConn).readLoop(0x400000c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 45 [select]: net/http.(*persistConn).writeLoop(0x400000c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 50 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400011a3e0, 0x12c18bc, 0x1, 0x4000315e58, 0x40001b6a80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x400001dec8, 0x40001b6a80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000315fa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4000182880) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 51 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4000182880) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 86 [IO wait]: internal/poll.runtime_pollWait(0xffffbc2282c0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000188518, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000188500, 0x4000464000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000188500, 0x4000464000, 0x1000, 0x1000, 0x400036abe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010578, 0x4000464000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400451c480, 0x4000464000, 0x1000, 0x1000, 0x400036acf8, 0x67867c, 0x40024634a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002465680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4002465680, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400451c480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 54 [IO wait]: internal/poll.runtime_pollWait(0xffffbc228cb8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000128098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000128080, 0x400007e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000128080, 0x400007e000, 0x1000, 0x1000, 0x400030dbe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000190068, 0x400007e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001ec480, 0x400007e000, 0x1000, 0x1000, 0x400030dcf8, 0x67867c, 0x40001b6d20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001b0840) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40001b0840, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40001ec480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 55 [select]: net/http.(*persistConn).writeLoop(0x40001ec480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 95 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x40024c10c0, 0x40000feee0, 0x40000fef50, 0x400047faf0, 0x12e8f90, 0x40043e22a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x400471df34, 0x5, 0x4000183d40, 0x12eca70, 0x4000183c00, 0x130d478, 0x40001de460, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40000e88c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40000e88c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 26 [select]: net/http.(*persistConn).writeLoop(0x400000d680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 48 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x40023ba540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 25 [select]: net/http.(*persistConn).readLoop(0x400000d680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 67 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x40023ba540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 68 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x40023ba540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 69 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x40023ba540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 65 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x40001a1680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 88 [IO wait]: internal/poll.runtime_pollWait(0xffffbc2281d8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000188b18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000188b00, 0x4000470000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000188b00, 0x4000470000, 0x1000, 0x1000, 0x400030f688, 0x69f178, 0x400030f6f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000105a0, 0x4000470000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400451c6c0, 0x4000470000, 0x1000, 0x1000, 0x5, 0x4002376f00, 0x400030f7c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40024659e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40024659e0, 0x68f90a, 0x400030f8a8, 0x412bc0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40024659e0, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000187bc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000187bc0, 0x4000472000, 0x1000, 0x1000, 0x400030f998, 0x44e884, 0x80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000182c40, 0x4000472000, 0x1000, 0x1000, 0x40003d7858, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000182c40, 0x4000472000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000182c80, 0x4000472000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400030fd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400030fd80, 0xa, 0x4000472201, 0x1, 0xffff9418f788, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400030fd80, 0x400047220a, 0x1, 0x40046298a0, 0x0, 0x4000472259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400030fd80, 0x40023f620a, 0x40046298a0, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4004454c70, 0x40004628c0, 0x1f, 0x400030fee0, 0x4000419f20, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x4004454c70, 0x113075a, 0x7, 0x40023614a0, 0xd, 0x4000419f20, 0x0, 0x0, 0x2) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x4004454c30, 0x40023614a0, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 70 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000081f80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 71 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x40024647e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 72 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x40004563f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 73 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x4000188100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 74 [IO wait]: internal/poll.runtime_pollWait(0xffffbc228578, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000188298, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000188280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000188280, 0x0, 0x4000428698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40024a1110, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40024a1110, 0x12ed9d0, 0x40024a1110, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x4000188200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 75 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4000099520, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 76 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4000099520, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 77 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x4000099520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 78 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x4000457260) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 87 [select]: net/http.(*persistConn).writeLoop(0x400451c480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 64 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x40001a1680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 81 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40001de0e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 63 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40024c0540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 98 [IO wait]: internal/poll.runtime_pollWait(0xffffbc228660, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000188a18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000188a00, 0x4000373000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000188a00, 0x4000373000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011e270, 0x4000373000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000d680, 0x4000373000, 0x1000, 0x1000, 0x40001bf500, 0x200000003, 0x40001bf500) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003483c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003483c0, 0x40001fc00a, 0x400009d8c8, 0x4d3b10, 0x40000820c0, 0x40001fc000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003483c0, 0x0, 0x1, 0x0, 0x40000820d8, 0x40001fc000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000186e40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000186e40, 0x400046e000, 0x1000, 0x1000, 0x400009d9b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000182940, 0x400046e000, 0x1000, 0x1000, 0x40003d7858, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000182940, 0x400046e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001829c0, 0x400046e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400009ddd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400009ddd0, 0xa, 0x400046e201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400009ddd0, 0x400046e20a, 0x1, 0x4002361489, 0x0, 0x400046e2e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400009ddd0, 0x4000099a0a, 0x4002361489, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4004454c70, 0x400012a6a8, 0x17, 0x400009df20, 0x40044b4750, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x4004454c70, 0x113075a, 0x7, 0x40044b4750, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x4004454c30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 94 [IO wait]: internal/poll.runtime_pollWait(0xffffbc228748, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000189318, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000189300, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000189300, 0x58ec48, 0x4002463740, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004770438, 0x4714f4, 0x4002463740, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004770438, 0x4002361500, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x4000183d40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 93 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000343280) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 15 [select]: net/http.(*persistConn).readLoop(0x40044a85a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 16 [select]: net/http.(*persistConn).writeLoop(0x40044a85a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 99 [IO wait]: internal/poll.runtime_pollWait(0xffffbc228490, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000128b98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000128b80, 0x40044bd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000128b80, 0x40044bd000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400440e0b0, 0x40044bd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40044a85a0, 0x40044bd000, 0x1000, 0x1000, 0x40001bfc80, 0x200000003, 0x40001bfc80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003d7740) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003d7740, 0x40001fc00a, 0x40003138c8, 0x4d3b10, 0x40000820c0, 0x40001fc000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003d7740, 0x0, 0x1, 0x0, 0x40000820d8, 0x40001fc000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000338ff0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000338ff0, 0x40043ac000, 0x1000, 0x1000, 0x40003139b8, 0x0, 0x4000313988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40003426c0, 0x40043ac000, 0x1000, 0x1000, 0x4000001c80, 0x3, 0x4000313be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40003426c0, 0x40043ac000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000342700, 0x40043ac000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000313dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000313dd0, 0xa, 0x40043ac201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000313dd0, 0x40043ac20a, 0x1, 0x40003338ab, 0x0, 0x40043ac263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000313dd0, 0x400034640a, 0x40003338ab, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4004454c70, 0x4000196900, 0x24, 0x4000313f20, 0x40044b4738, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x4004454c70, 0x113075a, 0x7, 0x40044b4738, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x4004454c30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 91 [select]: net/http.(*persistConn).readLoop(0x400451c6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 92 [select]: net/http.(*persistConn).writeLoop(0x400451c6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 100 [select]: net/http.(*persistConn).readLoop(0x400000d0e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 101 [select]: net/http.(*persistConn).writeLoop(0x400000d0e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 33 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40024c0780) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 125 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400011a3e0, 0x1144559, 0x1b, 0x4004dafe38, 0x400464de60, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x40046b1850, 0x400464de60, 0x40001ed0e0, 0x4000016640) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x12c2e00, 0x40046aff01) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004daff80, 0x12c2e00, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004e1c1e0, 0x40046b1850) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 124 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40023bad20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 152 [semacquire]: sync.runtime_Semacquire(0x4004e81f88) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004e81f80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x40001ce100, 0x4000186840, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffdcc51969, 0x29, 0xffffdcc519cb, 0x25, 0x40046d3da0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x400477ac20, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffdcc51969, 0x29, 0xffffdcc519cb, 0x25, 0x40046d3da0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x400477ac20, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004ddf190, 0x4004ddee60, 0x4000186840, 0x40001eb440, 0x4000081b80, 0x0, 0xffffbc224888, 0x78, 0xffffbc2a4f40, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x40001c0700, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x400477ac20, 0xb, 0x400477ac18, 0x6, 0x400477ac30, 0xd, 0x40047695c0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x40001c0700, 0x40003d0000, 0x40001873b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x40001c0700, 0x40024686f0, 0x2, 0x2, 0x4002468700, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 130 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x40024c10c0, 0x400047faf0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 131 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40043e22a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 132 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40043e22a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 133 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x40024bec60, 0x400464c9c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 134 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x40024c0500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 135 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x40001a1720) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 136 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x40001a1720) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 137 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x4000190508) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 138 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x40001a1720, 0x400464ca20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 139 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400011a3e0, 0x1144559, 0x1b, 0x400452de38, 0x400464cb40, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004d94610, 0x400464cb40, 0xf1f700, 0x400011bc60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400452df80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x400464f080, 0x4004d94610) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 140 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 141 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x4000192e60, 0xd00007, 0x4004d828c8, 0x5, 0x4004d94620, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 142 [IO wait]: internal/poll.runtime_pollWait(0xffffbc2280f0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004409418, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4004409400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4004409400, 0xffffbc2a4a68, 0x30, 0x400464b770) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004671f50, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004671f50, 0x400464b770, 0x11ae500, 0x40023bab00, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40023baa80, 0x12ed9d0, 0x4004671f50, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40023baa80, 0x12ed9d0, 0x4004671f50, 0x40001c0700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 143 [IO wait]: internal/poll.runtime_pollWait(0xffffbc228008, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004409518, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4004409500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4004409500, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004671f68, 0xffffbc2a4a68, 0xffff941311c0, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004671f68, 0x665e1c, 0x0, 0xfe5e60, 0x40043ae5c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004671f80, 0x400464b800, 0x11ae500, 0x40023babe0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40023bab60, 0x12ec710, 0x4004671f80, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40023bab60, 0x12ec710, 0x4004671f80, 0x40001c0700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 154 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40023bad20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 163 [IO wait]: internal/poll.runtime_pollWait(0xffff941afc00, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400037ce98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400037ce80, 0x4004ea4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400037ce80, 0x4004ea4000, 0x1000, 0x1000, 0x40044c5a48, 0x652ce0, 0x12edac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011e418, 0x4004ea4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001ed440, 0x4004ea4000, 0x1000, 0x1000, 0x113674e, 0x412bc0, 0x4000366050) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40046d8d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40046d8d80, 0xa, 0x0, 0x0, 0x0, 0x0, 0x400001dc38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40046d8d80, 0x0, 0x400001dc78, 0x655784, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400477e1b0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400477e1b0, 0x400476e800, 0x200, 0x200, 0x4, 0x0, 0x400001dd08) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000183ec0, 0x400476e800, 0x200, 0x200, 0x400003e407, 0x33, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000183ec0, 0x400476e800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000183f00, 0x400476e800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004731e40, 0x951400, 0x400001deb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004731e40, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004731e40, 0xee0d40, 0x400008cbe0, 0x12db068, 0x4000183f00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40047177a0, 0x40023d4d80, 0x40023d1ce0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 97 [IO wait]: internal/poll.runtime_pollWait(0xffff941afce8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000189418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000189400, 0x4004d30000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000189400, 0x4004d30000, 0x1000, 0x1000, 0x4000369be0, 0x469b10, 0x400467b980) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000108a8, 0x4004d30000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000133e60, 0x4004d30000, 0x1000, 0x1000, 0x4000369cf8, 0x67867c, 0x40023d1c80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40023d4c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40023d4c00, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4000133e60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 153 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40023bad20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 162 [select]: net/http.(*persistConn).writeLoop(0x4000133e60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 149 [IO wait]: internal/poll.runtime_pollWait(0xffffbc2283a8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400037c498, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400037c480, 0x40044cd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400037c480, 0x40044cd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400440e0f0, 0x40044cd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000d0e0, 0x40044cd000, 0x1000, 0x1000, 0x0, 0x0, 0x4004d93b28) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003d7f20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003d7f20, 0xa, 0x40043b23e8, 0x48731c, 0x9, 0x1, 0xf5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003d7f20, 0x0, 0x40043b2448, 0x57d438, 0x11320c7, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004dda180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004dda180, 0x4004653000, 0x200, 0x200, 0x4004d82948, 0x4004d82948, 0x4000196f90) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40024c1600, 0x4004653000, 0x200, 0x200, 0xa19fd7ea05, 0x41e4896, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40024c1600, 0x4004653000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40024c1640, 0x4004653000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004db06e0, 0x0, 0x40043b26b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004db06e0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004db06e0, 0xee0d40, 0x40001934a0, 0x12db068, 0x40024c1640) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004647830, 0x400464fc20, 0x400464d140) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 155 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40023bad20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 156 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40023bad20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 157 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x40024c1c80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 178 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40024c1cc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 158 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004e1e4d0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 159 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004e1e4d0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 160 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4004e1e4d0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 128 [select]: net/http.(*persistConn).readLoop(0x40001ed440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 129 [select]: net/http.(*persistConn).writeLoop(0x40001ed440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 179 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x4004e1e370) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 180 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x4004e1e370) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 181 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x4004e1e370) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 182 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x4004e1e370) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 183 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x40046d1400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 184 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x40046d1400, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 185 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x40001c0700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 186 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x40001c0700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 187 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x40001c0700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 188 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:50:19.294+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:50:19.294+00:00 [Info] Setting ipv6=false 2021-08-16T13:50:19.302+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:50:19.309+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:50:19.309+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:50:19.309+00:00 [Info] Setting log level to Info 2021-08-16T13:50:19.309+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:50:19.312+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:50:19 audit: created new audit service 2021-08-16T13:50:19.320+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:50:19.320+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:50:19.321+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:50:19.321+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:50:19.321+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:50:19.331+00:00 [Info] security context initialized 2021-08-16T13:50:19.331+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:50:19.331+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:50:19.331+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:50:19.331+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:50:19.331+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:50:19.348+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:50:19.349+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:50:19.349+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:50:19.370+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:50:19.372+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:50:19.373+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:50:19.373+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:50:19.372+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:50:19.374+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:50:19.375+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:50:19.377+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:50:19.422+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:50:19.422+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 51 us 2021-08-16T13:50:19.422+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:50:19.424+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:50:19.425+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:50:19.428+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:50:19.428+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:50:19.428+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:50:19.428+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:50:19.428+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:50:19.428+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:19.428+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:19.430+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:50:19.430+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:50:19.430+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:50:19.430+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:19.430+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:50:19.430+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:50:19.430+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:50:19.430+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:50:19.430+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:50:19.430+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:50:19.430+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:50:19.437+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 6.231806ms 2021-08-16T13:50:19.437+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:50:19.438+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:50:19.438+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:19.438+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:50:19.438+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:50:19.438+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:50:19.438+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:50:19.449+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:50:19.449+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:50:19.449+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:50:19.449+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards unexpected fault address 0x400015ad9f fatal error: fault 2021-08-16T13:50:19.449+00:00 [Info] : Recover shard shard1 from metadata completed. [signal SIGBUS: bus error code=0x1 addr=0x400015ad9f pc=0xbd8ea4] goroutine 198 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x40000909f0 sp=0x40000909c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4000090a30 sp=0x40000909f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x4004e10820, 0x4004ba5200, 0x11a9890, 0x4000090b40, 0x4004e10838, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4000090ab0 sp=0x4000090a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x4004e10820, 0x4004ba5200, 0x11a9890, 0x4000090b40, 0x4004e10838, 0xc18f8c, 0x4000090c18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4000090af0 sp=0x4000090ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4000032c80, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4000090da0 sp=0x4000090af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4000032c80, 0x40001c7730, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4000090e60 sp=0x4000090da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004e22b60, 0x4004e22b70, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4000091200 sp=0x4000090e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004e22b60, 0x4004e22b70, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x40000918f0 sp=0x4000091200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004e707e0, 0x4004e27500, 0x4000256680, 0x4004e22b40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4000091fb0 sp=0x40000918f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4000091fb0 sp=0x4000091fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x400018a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x40002016b0, 0x4, 0x1136a87, 0xf, 0x40001a1e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 34 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x40002120f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 35 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x4000212108) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 36 [IO wait]: internal/poll.runtime_pollWait(0xffff881bb758, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000198198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000198180, 0x40001a9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000198180, 0x40001a9000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400014a4f8, 0x40001a9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x400010e480, 0x40001ae001, 0xdff, 0xdff, 0x400012f8e8, 0x6902d8, 0x400019d948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400013c5b8, 0x40001ae001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x4000176840, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4000176840, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4000176840, 0xf5a960, 0x400010e560, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x400010e540, 0x4000152320, 0x400019db28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x4000118190, 0x12f8f98, 0x400010e540, 0x4000146ca7, 0x16, 0x4000157320, 0x400012fc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x4000118190, 0x12f8f98, 0x400010e540, 0x4000118190, 0x40001464b0, 0x40001464c0, 0x4000198200, 0x4000152240, 0xee0980, 0x4000157320, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x4000118190, 0x12f8f98, 0x400010e540) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4000200630, 0x400012ff78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400012df78, 0x4000200630, 0x12db8a8, 0x400013c588, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x4000200630, 0x4000222500, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x4000200630, 0x4000222500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 21 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400010c500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 37 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 38 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 39 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 40 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 41 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 42 [syscall]: syscall.Syscall(0x3f, 0x0, 0x4000247fa0, 0x4, 0x4000247e68, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x4000247fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x400010e060, 0x4000247fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x400014a000, 0x4000247fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 62 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x40002d2c80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 52 [select]: net/http.(*persistConn).readLoop(0x400000c240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 53 [select]: net/http.(*persistConn).writeLoop(0x400000c240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 46 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000204750, 0x12c18bc, 0x1, 0x40004b9e58, 0x400011a0c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x4000240ec8, 0x400011a0c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40004b9fa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4000208000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 47 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4000208000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 68 [IO wait]: internal/poll.runtime_pollWait(0xffff881bb670, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000198318, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000198300, 0x40001af000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000198300, 0x40001af000, 0x1000, 0x1000, 0x4000069a88, 0x68f7a4, 0x40001769f0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400014a0a8, 0x40001af000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c240, 0x40001af000, 0x1000, 0x1000, 0xf12180, 0x4000118450, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400010e5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400010e5a0, 0xa, 0x1008880, 0x4000118410, 0x199, 0xad9, 0x4000069b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400010e5a0, 0x2, 0x0, 0x9, 0x56b, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40001b4c90) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40001b4c90, 0x400006c000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000208300, 0x400006c000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000208300, 0x400006c000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000208340, 0x400006c000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40001769a0, 0x0, 0x4000069eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40001769a0, 0x0, 0x0, 0x4000069f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40001769a0, 0xee0d40, 0x4000118410, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4000186750, 0x40004835c0, 0x400011a840) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 66 [IO wait]: internal/poll.runtime_pollWait(0xffff881bb588, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000198418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000198400, 0x400001c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000198400, 0x400001c000, 0x1000, 0x1000, 0x40001a0be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400020c088, 0x400001c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c7e0, 0x400001c000, 0x1000, 0x1000, 0x40001a0cf8, 0x67867c, 0x400011a7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40004834a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40004834a0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000c7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 67 [select]: net/http.(*persistConn).writeLoop(0x400000c7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 73 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40002094c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 71 [select]: net/http.(*persistConn).readLoop(0x400000cea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 72 [select]: net/http.(*persistConn).writeLoop(0x400000cea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 30 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000208600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 27 [IO wait]: internal/poll.runtime_pollWait(0xffff881baf30, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000199918, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000199900, 0x4004408000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000199900, 0x4004408000, 0x1000, 0x1000, 0x400012f688, 0x69f178, 0x400012f6f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400020c3e0, 0x4004408000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000d9e0, 0x4004408000, 0x1000, 0x1000, 0x5, 0x4004466000, 0x400012f7c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000281080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000281080, 0x68f90a, 0x400016c790, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000281080, 0x0, 0x400012f868, 0x41a0a8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400444e690) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400444e690, 0x4004462000, 0x1000, 0x1000, 0x400012f998, 0x44e884, 0x1c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000208280, 0x4004462000, 0x1000, 0x1000, 0x400010f0d8, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000208280, 0x4004462000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000208380, 0x4004462000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400012fd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400012fd80, 0xa, 0x4004462201, 0x1, 0xffff8816e998, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400012fd80, 0x400446220a, 0x1, 0x40002af330, 0x0, 0x4004462259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400012fd80, 0x400017700a, 0x40002af330, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400018d560, 0x4004452000, 0x1f, 0x400012fee0, 0x4004450000, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x400018d560, 0x113075a, 0x7, 0x400009d4b0, 0xd, 0x4004450000, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x400018d520, 0x400009d4b0, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 5 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x4000190b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 63 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x40002d2c80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 75 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x4000190b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 6 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x4000190b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 7 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x4000190b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 78 [select]: net/http.(*persistConn).readLoop(0x400000d9e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 79 [select]: net/http.(*persistConn).writeLoop(0x400000d9e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 8 [IO wait]: internal/poll.runtime_pollWait(0xffff881bae48, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400020e698, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400020e680, 0x40001d8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400020e680, 0x40001d8000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400014a670, 0x40001d8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40000e2360, 0x40001d8000, 0x1000, 0x1000, 0x4000103500, 0x200000003, 0x4000103500) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400010fa40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400010fa40, 0x40001aa00a, 0x40004bb8c8, 0x4d3b10, 0x400010e0c0, 0x40001aa000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400010fa40, 0x0, 0x1, 0x0, 0x400010e0d8, 0x40001aa000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40002b4900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40002b4900, 0x4000455000, 0x1000, 0x1000, 0x40004bb9b8, 0x0, 0x40004bb988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40002ba400, 0x4000455000, 0x1000, 0x1000, 0x400023f380, 0x3, 0x40004bbbe0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40002ba400, 0x4000455000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40002ba440, 0x4000455000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40004bbdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40004bbdd0, 0xa, 0x4000455201, 0x1, 0xd20, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40004bbdd0, 0x400045520a, 0x1, 0x400009d499, 0x0, 0x40004552e0, 0x1, 0xd20, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40004bbdd0, 0x40000dc80a, 0x400009d499, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400018d560, 0x4000214ca8, 0x17, 0x40004bbf20, 0x40001c96b0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x400018d560, 0x113075a, 0x7, 0x40001c96b0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x400018d520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 9 [IO wait]: internal/poll.runtime_pollWait(0xffff881bb018, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000198e98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000198e80, 0x40004ea000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000198e80, 0x40004ea000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400020c248, 0x40004ea000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000cea0, 0x40004ea000, 0x1000, 0x1000, 0x4000103680, 0x200000003, 0x4000103680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40002104e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40002104e0, 0x40001aa00a, 0x40004b78c8, 0x4d3b10, 0x400010e0c0, 0x40001aa000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40002104e0, 0x0, 0x1, 0x0, 0x400010e0d8, 0x40001aa000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40000a28a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40000a28a0, 0x4002539000, 0x1000, 0x1000, 0x40004b79b8, 0x44d8a8, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40000ac4c0, 0x4002539000, 0x1000, 0x1000, 0x40000b2a18, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40000ac4c0, 0x4002539000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40000ac500, 0x4002539000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40004b7dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40004b7dd0, 0xa, 0x4002539201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40004b7dd0, 0x400253920a, 0x1, 0x400009d49c, 0x0, 0x4002539263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40004b7dd0, 0x400020a70a, 0x400009d49c, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400018d560, 0x4000206840, 0x24, 0x40004b7f20, 0x40001c9698, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x400018d560, 0x113075a, 0x7, 0x40001c9698, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x400018d520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 14 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40002098c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 12 [select]: net/http.(*persistConn).readLoop(0x40000e2360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 13 [select]: net/http.(*persistConn).writeLoop(0x40000e2360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 15 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x400010fce0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 16 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x4002516e40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 82 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x400020ec80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 83 [IO wait]: internal/poll.runtime_pollWait(0xffff881bac78, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400020ed98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400020ed80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400020ed80, 0x0, 0x4000242698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40001c9ba8, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40001c9ba8, 0x12ed9d0, 0x40001c9ba8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x400020ed00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 84 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400018dd40, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 85 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400018dd40, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 86 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x400018dd40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 87 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x4002517920) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 33 [IO wait]: internal/poll.runtime_pollWait(0xffff881bab90, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000e4e98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000e4e80, 0x4004624000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000e4e80, 0x4004624000, 0x1000, 0x1000, 0x4004411be0, 0x469b10, 0x4000295800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010140, 0x4004624000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40004c8b40, 0x4004624000, 0x1000, 0x1000, 0x4004411cf8, 0x67867c, 0x4004620240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40000b3740) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40000b3740, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40004c8b40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 115 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40000acfc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 90 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x400027e5b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 102 [select]: net/http.(*persistConn).readLoop(0x400000c480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 64 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x40002bb440, 0x40047646a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 103 [select]: net/http.(*persistConn).writeLoop(0x400000c480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 114 [select]: net/http.(*persistConn).writeLoop(0x40004c8b40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 125 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x40045ea630) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 124 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x40045ea630) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 123 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40024ffa40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 91 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40000ad840) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 92 [IO wait]: internal/poll.runtime_pollWait(0xffff881ba9c0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400020f418, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400020f400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400020f400, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40045df800, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40045df800, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x40024ff440) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 93 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x40002bb440, 0x40002bcd90, 0x40002bce00, 0x40047646a0, 0x12e8f90, 0x4002540b60, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x40045bf064, 0x5, 0x40024ff440, 0x12eca70, 0x40024ff300, 0x130d478, 0x400027e620, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40001907e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40001907e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 65 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x4002540b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 130 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x4002540b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 131 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x4002435170, 0x4004703560) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 132 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x4000208540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 133 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x40002d2d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 134 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x40002d2d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 135 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x4000286330) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 136 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x40002d2d20, 0x40047035c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 137 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000204750, 0x1144559, 0x1b, 0x4000093e38, 0x40047036e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x40047651c0, 0x40047036e0, 0xf1f700, 0x4000205fd0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000093f80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004c625a0, 0x40047651c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 138 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 139 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400028eb40, 0xd00007, 0x4004767cf8, 0x5, 0x40047651d0, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 140 [IO wait]: internal/poll.runtime_pollWait(0xffff881bad60, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40002e1098, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40002e1080, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40002e1080, 0xffffb007ff18, 0x30, 0x4004c23110) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c172a8, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c172a8, 0x4004c23110, 0x11ae500, 0x4004510780, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x4004510700, 0x12ed9d0, 0x4004c172a8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x4004510700, 0x12ed9d0, 0x4004c172a8, 0x400018a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 141 [IO wait]: internal/poll.runtime_pollWait(0xffff881ba8d8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40002e1198, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40002e1180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40002e1180, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c172c0, 0xffffb007ff18, 0xffff8804f278, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c172c0, 0x665e1c, 0x0, 0xfe5e60, 0x40002ee5c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004c172d8, 0x4004c231a0, 0x11ae500, 0x4004510860, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40045107e0, 0x12ec710, 0x4004c172d8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40045107e0, 0x12ec710, 0x4004c172d8, 0x400018a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 168 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40001909a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 167 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40001909a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 95 [IO wait]: internal/poll.runtime_pollWait(0xffff88197a00, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400020f618, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400020f600, 0x4004d2c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400020f600, 0x4004d2c000, 0x1000, 0x1000, 0x400019bbe0, 0x469b10, 0x4004c5f800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400014aa20, 0x4004d2c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004c54a20, 0x4004d2c000, 0x1000, 0x1000, 0x400019bcf8, 0x67867c, 0x400011baa0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40045e8360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40045e8360, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004c54a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 96 [select]: net/http.(*persistConn).writeLoop(0x4004c54a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 122 [IO wait]: internal/poll.runtime_pollWait(0xffff88197918, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400020fa18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400020fa00, 0x4004ea8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400020fa00, 0x4004ea8000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400014ab50, 0x4004ea8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40000e3440, 0x4004ea8000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40045e97a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40045e97a0, 0xa, 0x4002410123, 0x11320c7, 0x9, 0x1, 0x4000058457) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40045e97a0, 0xd4fa28, 0x400003e280, 0x46, 0x11320c7, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004be4a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004be4a20, 0x4004bd5c00, 0x200, 0x200, 0x400467f8c0, 0x400467f8c0, 0x40000a6bd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000169f80, 0x4004bd5c00, 0x200, 0x200, 0x40000a7f87, 0xe, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000169f80, 0x4004bd5c00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000169fc0, 0x4004bd5c00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004e1edc0, 0x680d00, 0x40000586b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004e1edc0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004e1edc0, 0xee0d40, 0x40000e8e60, 0x12db068, 0x4000169fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400485ddd0, 0x4004857380, 0x4004bd9200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 166 [semacquire]: sync.runtime_Semacquire(0x4004e707e8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004e707e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x4000256680, 0x40001b4810, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffc6ebd969, 0x29, 0xffffc6ebd9cb, 0x25, 0x4004bd9800, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x40045bfd50, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffc6ebd969, 0x29, 0xffffc6ebd9cb, 0x25, 0x4004bd9800, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x40045bfd50, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4000091190, 0x4000090e60, 0x40001b4810, 0x4000273b00, 0x40002094c0, 0x0, 0xffff8803bd10, 0x88, 0xffffb007f5e0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x400018a000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x40045bfd50, 0xb, 0x40045bfd48, 0x6, 0x40045bfd60, 0xd, 0x40045e08c0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x400018a000, 0x4000480000, 0x40004be570) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x400018a000, 0x4000285090, 0x2, 0x2, 0x40002850a0, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 147 [IO wait]: internal/poll.runtime_pollWait(0xffff881baaa8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000198818, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000198800, 0x4004458000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000198800, 0x4004458000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400020c070, 0x4004458000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c480, 0x4004458000, 0x1000, 0x1000, 0x0, 0x0, 0x4004c6bb28) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000281860) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000281860, 0xa, 0x0, 0x0, 0x0, 0x0, 0x40002ee418) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000281860, 0x412bc0, 0x0, 0x0, 0x40002ee4c8, 0x412bc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004c23a10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004c23a10, 0x4004c7e200, 0x200, 0x200, 0x4004c23170, 0x30, 0x40002ee578) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40002bb980, 0x4004c7e200, 0x200, 0x200, 0x0, 0xffffb007ff18, 0x40002ee568) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40002bb980, 0x4004c7e200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40002bb9c0, 0x4004c7e200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004c1da20, 0x12ec700, 0x40002ee6b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004c1da20, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004c1da20, 0xee0d40, 0x400028f0e0, 0x12db068, 0x40002bb9c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400474fb90, 0x4004c630e0, 0x4004703c80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 169 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40001909a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 170 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40001909a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 171 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x40024ffa00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 183 [IO wait]: internal/poll.runtime_pollWait(0xffff88197ae8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400020f898, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400020f880, 0x4004eb2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400020f880, 0x4004eb2000, 0x1000, 0x1000, 0x4004eb0be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400014ab58, 0x4004eb2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40000e3320, 0x4004eb2000, 0x1000, 0x1000, 0x4004eb0cf8, 0x67867c, 0x4004d35500) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40045e98c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40045e98c0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40000e3320) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 174 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40045ea790, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 175 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40045ea790, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 176 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x40045ea790) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 177 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40001909a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 178 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000204750, 0x1144559, 0x1b, 0x40004b5e38, 0x4004d347e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x40045c1020, 0x4004d347e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40004b5f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40045e8ae0, 0x40045c1020) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 184 [select]: net/http.(*persistConn).writeLoop(0x40000e3320) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 181 [select]: net/http.(*persistConn).readLoop(0x40000e3440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 182 [select]: net/http.(*persistConn).writeLoop(0x40000e3440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 126 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x40045ea630) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 127 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x40045ea630) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 128 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004e60200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 129 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x4004e60200, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 194 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x400018a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 195 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x400018a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 196 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x400018a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 197 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:50:24.608+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:50:24.609+00:00 [Info] Setting ipv6=false 2021-08-16T13:50:24.626+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:50:24.629+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:50:24.629+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:50:24.629+00:00 [Info] Setting log level to Info 2021-08-16T13:50:24.629+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:50:24.631+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:50:24 audit: created new audit service 2021-08-16T13:50:24.641+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:50:24.641+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:50:24.641+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:50:24.642+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:50:24.642+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:50:24.652+00:00 [Info] security context initialized 2021-08-16T13:50:24.652+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:50:24.652+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:50:24.652+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:50:24.652+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:50:24.652+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:50:24.664+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:50:24.665+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:50:24.678+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:50:24.691+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:50:24.691+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:50:24.692+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:50:24.693+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:50:24.694+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:50:24.694+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:50:24.695+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:50:24.695+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:50:24.745+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:50:24.746+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 49 us 2021-08-16T13:50:24.746+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:50:24.748+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:50:24.748+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:50:24.750+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:50:24.750+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:50:24.750+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:50:24.750+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:50:24.750+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:50:24.752+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:24.752+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:24.753+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:50:24.753+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:50:24.753+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:50:24.754+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:24.755+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:50:24.755+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:50:24.755+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:50:24.755+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:50:24.755+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:50:24.755+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:50:24.755+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:50:24.760+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 4.65265ms 2021-08-16T13:50:24.760+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:50:24.761+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:50:24.761+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:24.761+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:50:24.761+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:50:24.761+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:50:24.761+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:50:24.772+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:50:24.772+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:50:24.772+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:50:24.773+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:50:24.773+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x40047ca91f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x40047ca91f pc=0xbd8ea4] goroutine 186 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004d6c9f0 sp=0x4004d6c9c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004d6ca30 sp=0x4004d6c9f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x4004ec4000, 0x40047e1800, 0x11a9890, 0x4004d6cb40, 0x4004ec4018, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004d6cab0 sp=0x4004d6ca40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x4004ec4000, 0x40047e1800, 0x11a9890, 0x4004d6cb40, 0x4004ec4018, 0xffffa0275448, 0x4004d6cc18, 0x49) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004d6caf0 sp=0x4004d6cab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4000033040, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004d6cda0 sp=0x4004d6caf0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4000033040, 0x40001e1650, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004d6ce60 sp=0x4004d6cda0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004cc8e00, 0x4004cc8e10, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004d6d200 sp=0x4004d6ce60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004cc8e00, 0x4004cc8e10, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004d6d8f0 sp=0x4004d6d200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004eb28a0, 0x4004cd9180, 0x4004471600, 0x4004cc8de0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004d6dfb0 sp=0x4004d6d8f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004d6dfb0 sp=0x4004d6dfb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x40000e2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x4000111770, 0x4, 0x1136a87, 0xf, 0x4000119e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 18 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400011a0f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 19 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400011a108) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 20 [IO wait]: internal/poll.runtime_pollWait(0xffffa0287108, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000150718, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000150700, 0x40000f2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000150700, 0x40000f2000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011e0a0, 0x40000f2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x40000823c0, 0x40000f3001, 0xdff, 0xdff, 0x400009f8e8, 0x6902d8, 0x4000115948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400000e588, 0x40000f3001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40000ce840, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40000ce840, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40000ce840, 0xf5a960, 0x40000824a0, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4000082480, 0x4000046340, 0x4000115b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400008c190, 0x12f8f98, 0x4000082480, 0x4000036ca7, 0x16, 0x400000d0e0, 0x400009fc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400008c190, 0x12f8f98, 0x4000082480, 0x400008c190, 0x40000364b8, 0x40000364c0, 0x40000f0100, 0x4000046260, 0xee0980, 0x400000d0e0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400008c190, 0x12f8f98, 0x4000082480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4000110570, 0x400009ff78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400009bf78, 0x4000110570, 0x12db8a8, 0x400011a120, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x4000110570, 0x4000134320, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x4000110570, 0x4000134320) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 21 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 22 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 34 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x4000490000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 23 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 24 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 25 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 50 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40000557a0, 0x4, 0x4000055668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40000557a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x40000557a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x40000557a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 54 [IO wait]: internal/poll.runtime_pollWait(0xffffa0287020, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0200, 0x4000012000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0200, 0x4000012000, 0x1000, 0x1000, 0x4000066a88, 0x68f7a4, 0x4000130890) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400018e010, 0x4000012000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c5a0, 0x4000012000, 0x1000, 0x1000, 0xf12180, 0x400012a1d0, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400038ec00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400038ec00, 0xa, 0x1008880, 0x400012a190, 0x199, 0x1000, 0x4000066b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400038ec00, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40001828d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40001828d0, 0x40000d2000, 0xe00, 0xe00, 0x4000066da8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000126080, 0x40000d2000, 0xe00, 0xe00, 0x4000066e08, 0x40bb94, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000126080, 0x40000d2000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001260c0, 0x40000d2000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4000130840, 0x0, 0x4000066eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4000130840, 0x0, 0x0, 0x4000066f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4000130840, 0xee0d40, 0x400012a190, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4000016000, 0x400038ef60, 0x40001a42a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 52 [select]: net/http.(*persistConn).readLoop(0x400000c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 53 [select]: net/http.(*persistConn).writeLoop(0x400000c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 28 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400011c7d0, 0x12c18bc, 0x1, 0x4000413e58, 0x4000038120, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x400049eec8, 0x4000038120, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000413fa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4000080000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 29 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4000080000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 70 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000126380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 32 [IO wait]: internal/poll.runtime_pollWait(0xffffa0286f38, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0318, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0300, 0x40000d0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0300, 0x40000d0000, 0x1000, 0x1000, 0x4000488be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010378, 0x40000d0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000cb40, 0x40000d0000, 0x1000, 0x1000, 0x4000488cf8, 0x67867c, 0x4000038720) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40004808a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40004808a0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000cb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 33 [select]: net/http.(*persistConn).writeLoop(0x400000cb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 156 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40024c48c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 155 [semacquire]: sync.runtime_Semacquire(0x4004eb28a8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004eb28a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x4004471600, 0x4000400720, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xfffffec7c969, 0x29, 0xfffffec7c9cb, 0x25, 0x4004cde7e0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x40048530c0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xfffffec7c969, 0x29, 0xfffffec7c9cb, 0x25, 0x4004cde7e0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x40048530c0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004d6d190, 0x4004d6ce60, 0x4000400720, 0x400017cb40, 0x4000126380, 0x0, 0x3f, 0x4000448da8, 0xffff7811e5b0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x40000e2000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x40048530c0, 0xb, 0x40048530b8, 0x6, 0x40048530d0, 0xd, 0x400484a3c0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x40000e2000, 0x4000180000, 0x40003be1b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x40000e2000, 0x4000186ec0, 0x2, 0x2, 0x4000186ed0, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 40 [IO wait]: internal/poll.runtime_pollWait(0xffffa0286628, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000150618, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000150600, 0x40024dc000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000150600, 0x40024dc000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400018e0c0, 0x40024dc000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000418120, 0x40024dc000, 0x1000, 0x1000, 0x4000497500, 0x200000003, 0x4000497500) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000480d20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000480d20, 0x400011200a, 0x400009d8c8, 0x4d3b10, 0x40000820c0, 0x4000112000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000480d20, 0x0, 0x1, 0x0, 0x40000820d8, 0x4000112000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4002497080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4002497080, 0x400044f000, 0x1000, 0x1000, 0x400009d9b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400018a8c0, 0x400044f000, 0x1000, 0x1000, 0x40000837f8, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400018a8c0, 0x400044f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400018a900, 0x400044f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400009ddd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400009ddd0, 0xa, 0x400044f201, 0x1, 0xd20, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400009ddd0, 0x400044f20a, 0x1, 0x4004400d29, 0x0, 0x400044f2e0, 0x1, 0xd20, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400009ddd0, 0x4000128c0a, 0x4004400d29, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000e42b0, 0x40024bc168, 0x17, 0x400009df20, 0x40023f0390, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x40000e42b0, 0x113075a, 0x7, 0x40023f0390, 0x0, 0x1, 0x4000036bb0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x40000e4270) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 72 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x40000e8a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 82 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000126c40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 44 [select]: net/http.(*persistConn).readLoop(0x4000418120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 38 [select]: net/http.(*persistConn).readLoop(0x400000d8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 76 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x40000e8a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 77 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x40000e8a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 78 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x40000e8a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 16 [IO wait]: internal/poll.runtime_pollWait(0xffffa0286458, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0e98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0e80, 0x4000451000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0e80, 0x4000451000, 0x1000, 0x1000, 0x400044b688, 0x69f178, 0x400044b6f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011e140, 0x4000451000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400017ea20, 0x4000451000, 0x1000, 0x1000, 0x5, 0x40004a6f00, 0x400044b7c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000083aa0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000083aa0, 0x68f90a, 0x400030ae70, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000083aa0, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40043d0ab0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40043d0ab0, 0x4004420000, 0x1000, 0x1000, 0x400044b998, 0x44e884, 0x400044b948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000126480, 0x4004420000, 0x1000, 0x1000, 0x4000480718, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000126480, 0x4004420000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000126500, 0x4004420000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400044bd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400044bd80, 0xa, 0x4004420201, 0x1, 0xffffa029ba98, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400044bd80, 0x400442020a, 0x1, 0x400444a1c0, 0x0, 0x4004420259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400044bd80, 0x4000130f0a, 0x400444a1c0, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000e42b0, 0x40024424a0, 0x1f, 0x400044bee0, 0x40023e6a20, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x40000e42b0, 0x113075a, 0x7, 0x4004400d40, 0xd, 0x40023e6a20, 0x0, 0x0, 0x2) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x40000e4270, 0x4004400d40, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 39 [select]: net/http.(*persistConn).writeLoop(0x400000d8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 41 [IO wait]: internal/poll.runtime_pollWait(0xffffa02868e0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1918, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f1900, 0x40024a0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f1900, 0x40024a0000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011e4c8, 0x40024a0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000d8c0, 0x40024a0000, 0x1000, 0x1000, 0x4000103200, 0x200000003, 0x4000103200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000083320) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000083320, 0x400011200a, 0x40024db8c8, 0x4d3b10, 0x40000820c0, 0x4000112000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000083320, 0x0, 0x1, 0x0, 0x40000820d8, 0x4000112000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4002496a50) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4002496a50, 0x4000438000, 0x1000, 0x1000, 0x40024db9b8, 0x0, 0x40024db988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400018a440, 0x4000438000, 0x1000, 0x1000, 0x4000496d80, 0x3, 0x40024dbbe0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400018a440, 0x4000438000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400018a480, 0x4000438000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40024dbdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40024dbdd0, 0xa, 0x4000438201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40024dbdd0, 0x400043820a, 0x1, 0x400440099b, 0x0, 0x4000438263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40024dbdd0, 0x400018c30a, 0x400440099b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000e42b0, 0x40024ba2d0, 0x24, 0x40024dbf20, 0x40023f0378, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x40000e42b0, 0x113075a, 0x7, 0x40023f0378, 0x0, 0x4000189e78, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x40000e4270) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 83 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x40001808a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 84 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x4004404720) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 85 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x4000118080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 86 [IO wait]: internal/poll.runtime_pollWait(0xffffa02867f8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000118198, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000118180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000118180, 0x4714f4, 0x4000058e98, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40023e10c8, 0x4000058f00, 0x0, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40023e10c8, 0x12ed9d0, 0x40023e10c8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x4000118100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 87 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40001b0a90, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 88 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40001b0a90, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 89 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x40001b0a90) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 90 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x4004405200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 96 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000127d80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 59 [IO wait]: internal/poll.runtime_pollWait(0xffff7811fdc0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000197418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000197400, 0x4004c93000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000197400, 0x4004c93000, 0x1000, 0x1000, 0x400048ebe0, 0x469b10, 0x40001f6900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010740, 0x4004c93000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40043f6fc0, 0x4004c93000, 0x1000, 0x1000, 0x400048ecf8, 0x67867c, 0x40044598c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400441dbc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x400441dbc0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40043f6fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 93 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40000e0150) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 157 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40024c48c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 45 [select]: net/http.(*persistConn).writeLoop(0x4000418120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 56 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000127500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 10 [select]: net/http.(*persistConn).readLoop(0x400017e6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 11 [select]: net/http.(*persistConn).writeLoop(0x400017e6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 102 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x40001aeb40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 100 [select]: net/http.(*persistConn).readLoop(0x400017ea20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 101 [select]: net/http.(*persistConn).writeLoop(0x400017ea20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 55 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400018a7c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 97 [IO wait]: internal/poll.runtime_pollWait(0xffffa0286710, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000118b98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000118b80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000118b80, 0x58ec48, 0x40001a4b40, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x400484e528, 0x4714f4, 0x40001a4b40, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x400484e528, 0x40043cfc00, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x400440bb40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 114 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x400018b800, 0x400440cfc0, 0x400440d030, 0x40045026b0, 0x12e8f90, 0x40000e8d20, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x40048523d4, 0x5, 0x400440bb40, 0x12eca70, 0x400440ba00, 0x130d478, 0x40000e0380, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40043fc7e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40043fc7e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 103 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x40001aeb40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 104 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x400018b800, 0x40045026b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 105 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40000e8d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 106 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40000e8d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 107 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x400249ae10, 0x40004211a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 108 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x400018a780) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 109 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x40001aebe0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 110 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x40001aebe0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 111 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x400011e538) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 112 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x40001aebe0, 0x4000421200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 113 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400011c7d0, 0x1144559, 0x1b, 0x40024d9e38, 0x4000421320, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004502c10, 0x4000421320, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40024d9f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4002406a20, 0x4004502c10) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 130 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 131 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400008cb40, 0xd00007, 0x400454a4d0, 0x5, 0x4004502c20, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 132 [IO wait]: internal/poll.runtime_pollWait(0xffffa0286370, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1298, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f1280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f1280, 0xffffa02f5108, 0x30, 0x400455a0c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004504918, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004504918, 0x400455a0c0, 0x11ae500, 0x40004344e0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x4000434460, 0x12ed9d0, 0x4004504918, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x4000434460, 0x12ed9d0, 0x4004504918, 0x40000e2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 133 [IO wait]: internal/poll.runtime_pollWait(0xffffa0286288, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1b18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f1b00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f1b00, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004504930, 0xffffa02f5108, 0xffff7811d0f8, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004504930, 0x665e1c, 0x0, 0xfe5e60, 0x40004425c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004504948, 0x400455a150, 0x11ae500, 0x40004345c0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x4000434540, 0x12ec710, 0x4004504948, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x4000434540, 0x12ec710, 0x4004504948, 0x40000e2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 158 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40024c48c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 151 [IO wait]: internal/poll.runtime_pollWait(0xffff7811fcd8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1c98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f1c80, 0x40046a9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f1c80, 0x40046a9000, 0x1000, 0x1000, 0x4004514be0, 0x469b10, 0x40001f6d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400018e220, 0x40046a9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000418ea0, 0x40046a9000, 0x1000, 0x1000, 0x4004514cf8, 0x67867c, 0x4004564120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40046b0f00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40046b0f00, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4000418ea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 62 [IO wait]: internal/poll.runtime_pollWait(0xffff7811fbf0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000151b98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000151b80, 0x4004d81000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000151b80, 0x4004d81000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400018e368, 0x4004d81000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40004190e0, 0x4004d81000, 0x1000, 0x1000, 0x20, 0x30, 0x20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004dd42a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004dd42a0, 0xa, 0x0, 0x0, 0x0, 0x0, 0x4004cd5c38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004dd42a0, 0x0, 0x4004cd5c78, 0x655784, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40047b7590) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40047b7590, 0x4004cd6200, 0x200, 0x200, 0x4, 0x0, 0x4004cd5d08) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40047f2640, 0x4004cd6200, 0x200, 0x200, 0x4004c9a207, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40047f2640, 0x4004cd6200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40047f2680, 0x4004cd6200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004ccb4a0, 0x0, 0x4004cd5eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004ccb4a0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004ccb4a0, 0xee0d40, 0x4000484cd0, 0x12db068, 0x40047f2680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40047dcf30, 0x400441dec0, 0x4004459f20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 134 [IO wait]: internal/poll.runtime_pollWait(0xffffa0286540, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0a18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0a00, 0x4000439000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0a00, 0x4000439000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011e098, 0x4000439000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400017e6c0, 0x4000439000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000082d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000082d80, 0xa, 0x0, 0x0, 0x0, 0x0, 0x4000442418) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000082d80, 0x412bc0, 0x0, 0x0, 0x40004424c8, 0x412bc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400455a1b0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400455a1b0, 0x4002441000, 0x200, 0x200, 0x400455a120, 0x30, 0x4000442578) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400018bb80, 0x4002441000, 0x200, 0x200, 0x0, 0xffffa02f5108, 0x4000442568) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400018bb80, 0x4002441000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400018bbc0, 0x4002441000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004512840, 0x12ec700, 0x40004426b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004512840, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004512840, 0xee0d40, 0x400008ccd0, 0x12db068, 0x400018bbc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4000017170, 0x4002407020, 0x4004564000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 152 [select]: net/http.(*persistConn).writeLoop(0x4000418ea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 60 [select]: net/http.(*persistConn).writeLoop(0x40043f6fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 159 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40024c48c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 160 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x400469ad80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 172 [IO wait]: internal/poll.runtime_pollWait(0xffffa02869c8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000151a98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000151a80, 0x4004dd7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000151a80, 0x4004dd7000, 0x1000, 0x1000, 0x4004560be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400018e370, 0x4004dd7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000418fc0, 0x4004dd7000, 0x1000, 0x1000, 0x4004560cf8, 0x67867c, 0x4004dc2d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004dd43c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004dd43c0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4000418fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 163 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40024b9b80, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 164 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40024b9b80, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 165 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x40024b9b80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 166 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40024c48c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 167 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400011c7d0, 0x1144559, 0x1b, 0x4000445e38, 0x4004dc2060, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x40046d2c90, 0x4004dc2060, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000445f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40046b1680, 0x40046d2c90) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 173 [select]: net/http.(*persistConn).writeLoop(0x4000418fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 170 [select]: net/http.(*persistConn).readLoop(0x40004190e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 171 [select]: net/http.(*persistConn).writeLoop(0x40004190e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 63 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400469adc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 64 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x40024b9a20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 65 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x40024b9a20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 178 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x40024b9a20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 179 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x40024b9a20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 180 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x40047bf800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 181 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x40047bf800, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 182 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x40000e2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 183 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x40000e2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 184 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x40000e2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 185 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:50:29.924+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:50:29.925+00:00 [Info] Setting ipv6=false 2021-08-16T13:50:29.942+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:50:29.943+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:50:29.943+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:50:29.943+00:00 [Info] Setting log level to Info 2021-08-16T13:50:29.944+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:50:29.947+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:50:29 audit: created new audit service 2021-08-16T13:50:29.953+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:50:29.953+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:50:29.953+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:50:29.953+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:50:29.953+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:50:29.963+00:00 [Info] security context initialized 2021-08-16T13:50:29.963+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:50:29.963+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:50:29.963+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:50:29.963+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:50:29.963+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:50:29.976+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:50:29.982+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:50:29.990+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:50:30.006+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:50:30.006+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:50:30.007+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:50:30.010+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:50:30.010+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:50:30.010+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:50:30.010+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:50:30.015+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:50:30.063+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:50:30.063+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 66 us 2021-08-16T13:50:30.063+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:50:30.065+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:50:30.066+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:50:30.067+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:50:30.067+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:50:30.067+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:50:30.067+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:50:30.067+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:50:30.069+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:30.069+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:30.070+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:50:30.070+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:50:30.070+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:50:30.070+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:30.071+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:50:30.071+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:50:30.071+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:50:30.071+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:50:30.071+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:50:30.071+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:50:30.071+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:50:30.076+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 4.349178ms 2021-08-16T13:50:30.076+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:50:30.077+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:50:30.077+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:30.077+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:50:30.077+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:50:30.077+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:50:30.077+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:50:30.089+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:50:30.089+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:50:30.089+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:50:30.089+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:50:30.089+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x40047546df fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x40047546df pc=0xbd8ea4] goroutine 162 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004da09f0 sp=0x4004da09c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004da0a30 sp=0x4004da09f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x40003f9860, 0x4004e8e600, 0x11a9890, 0x4004da0b40, 0x40003f9878, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004da0ab0 sp=0x4004da0a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x40003f9860, 0x4004e8e600, 0x11a9890, 0x4004da0b40, 0x40003f9878, 0xffff993ad6b0, 0x4004da0c18, 0x49) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004da0af0 sp=0x4004da0ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x40046b0780, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004da0da0 sp=0x4004da0af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x40046b0780, 0x4000152f50, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004da0e60 sp=0x4004da0da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004c175c0, 0x4004c175d0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004da1200 sp=0x4004da0e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004c175c0, 0x4004c175d0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004da18f0 sp=0x4004da1200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004c6df30, 0x4004ea0e00, 0x40046ab080, 0x4004c175a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004da1fb0 sp=0x4004da18f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004da1fb0 sp=0x4004da1fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x40000ea000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x4000180db0, 0x4, 0x1136a87, 0xf, 0x4000107e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 18 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400012c108) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 19 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400012c120) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 20 [IO wait]: internal/poll.runtime_pollWait(0xffff7006ce88, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000190118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000190100, 0x40001ab000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000190100, 0x40001ab000, 0x1000, 0x1000, 0x1311860, 0x47b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000188010, 0x40001ab000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x40001861e0, 0x4000302001, 0xdff, 0xdff, 0x40000a38e8, 0x6902d8, 0x4000103948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x4000192030, 0x4000302001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x4000136160, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4000136160, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4000136160, 0xf5a960, 0x4000116860, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4000116840, 0x400011e540, 0x4000103b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x4000132690, 0x12f8f98, 0x4000116840, 0x4000124fb7, 0x16, 0x4000304120, 0x40000a3c01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x4000132690, 0x12f8f98, 0x4000116840, 0x4000132690, 0x40001247c8, 0x40001247d0, 0x400012e700, 0x400011e460, 0xee0980, 0x4000304120, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x4000132690, 0x12f8f98, 0x4000116840) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4000118630, 0x40000a3f78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x40004e3f78, 0x4000118630, 0x12db8a8, 0x4000192000, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x4000118630, 0x400013a460, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x4000118630, 0x400013a460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 34 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 35 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 36 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 37 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 38 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 50 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400040a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 39 [syscall]: syscall.Syscall(0x3f, 0x0, 0x400049a7a0, 0x4, 0x400049a668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x400049a7a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x400049a7a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x400049a7a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 55 [IO wait]: internal/poll.runtime_pollWait(0xffff7006cda0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400012e198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400012e180, 0x4000303000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400012e180, 0x4000303000, 0x1000, 0x1000, 0x400031ca88, 0x68f7a4, 0x4000136e10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000100e0, 0x4000303000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c5a0, 0x4000303000, 0x1000, 0x1000, 0xf12180, 0x400008c220, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000082600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000082600, 0xa, 0x1008880, 0x400008c1e0, 0x199, 0xad9, 0x400031cb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000082600, 0x2, 0x0, 0x9, 0x56b, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000450300) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000450300, 0x4000441000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000081880, 0x4000441000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000081880, 0x4000441000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40000818c0, 0x4000441000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4000136dc0, 0x0, 0x400031ceb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4000136dc0, 0x0, 0x0, 0x400031cf70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4000136dc0, 0xee0d40, 0x400008c1e0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40000e6a20, 0x4000082b40, 0x40000388a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 53 [select]: net/http.(*persistConn).readLoop(0x400000c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 54 [select]: net/http.(*persistConn).writeLoop(0x400000c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 24 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001843a0, 0x12c18bc, 0x1, 0x40000a9e58, 0x40000381e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x40004956c8, 0x40000381e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40000a9fa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4000080000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 25 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4000080000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 99 [select]: net/http.(*persistConn).writeLoop(0x4000304c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 28 [IO wait]: internal/poll.runtime_pollWait(0xffff7006ccb8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000190098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000190080, 0x4000346000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000190080, 0x4000346000, 0x1000, 0x1000, 0x4000065be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011c070, 0x4000346000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000304360, 0x4000346000, 0x1000, 0x1000, 0x4000065cf8, 0x67867c, 0x400014c480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000404660) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000404660, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4000304360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 29 [select]: net/http.(*persistConn).writeLoop(0x4000304360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 12 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000081e40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 64 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x40001aebe0, 0x40047568a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 90 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40004104c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 62 [select]: net/http.(*persistConn).writeLoop(0x4000304000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 45 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x4002428380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 115 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x40004009b0, 0xd00007, 0x40047471e0, 0x5, 0x4004c162c0, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 114 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 56 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x4002428380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 116 [IO wait]: internal/poll.runtime_pollWait(0xffff7006c0f0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400012eb18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400012eb00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400012eb00, 0xffff993a15b8, 0x30, 0x4004705650) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x400477fba8, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x400477fba8, 0x4004705650, 0x11ae500, 0x40046ae4e0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40046ae460, 0x12ed9d0, 0x400477fba8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40046ae460, 0x12ed9d0, 0x400477fba8, 0x40000ea000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 46 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x4002428380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 47 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x4002428380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 59 [select]: net/http.(*persistConn).readLoop(0x4002540000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 60 [select]: net/http.(*persistConn).writeLoop(0x4002540000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 14 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400018e780) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 15 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x40003c5080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 16 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x40004f6690) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 66 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x4000190000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 67 [IO wait]: internal/poll.runtime_pollWait(0xffff7006c578, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000190818, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000190800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000190800, 0x0, 0x4000054698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40000185b8, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40000185b8, 0x12ed9d0, 0x40000185b8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x4000190680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 68 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400441c680, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 69 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400441c680, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 70 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x400441c680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 71 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x40004f7170) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 65 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001843a0, 0x1144559, 0x1b, 0x400034fe38, 0x40047569c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004c162b0, 0x40047569c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0xdf79d0, 0x40003a9748) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400034ff80, 0x10, 0x12daa28) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004c18240, 0x4004c162b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 61 [select]: net/http.(*persistConn).readLoop(0x4000304000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 48 [IO wait]: internal/poll.runtime_pollWait(0xffff7006c3a8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000190a18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000190a00, 0x400001d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000190a00, 0x400001d000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400440c0b8, 0x400001d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000304120, 0x400001d000, 0x1000, 0x1000, 0x400010af00, 0x200000003, 0x400010af00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003c5740) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003c5740, 0x400045c00a, 0x40004e58c8, 0x4d3b10, 0x40000820c0, 0x400045c000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003c5740, 0x0, 0x1, 0x0, 0x40000820d8, 0x400045c000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40004f75f0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40004f75f0, 0x40000ff000, 0x1000, 0x1000, 0x40004e59b8, 0x0, 0x40004e5988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400440e480, 0x40000ff000, 0x1000, 0x1000, 0x400010a900, 0x3, 0x40004e5be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400440e480, 0x40000ff000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400440e4c0, 0x40000ff000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40004e5dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40004e5dd0, 0xa, 0x40000ff201, 0x1, 0xd1f, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40004e5dd0, 0x40000ff20a, 0x1, 0x40004f2f99, 0x0, 0x40000ff2e1, 0x1, 0xd1f, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40004e5dd0, 0x400441c70a, 0x40004f2f99, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000a1a40, 0x40004f4228, 0x17, 0x40004e5f20, 0x4000018600, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x40000a1a40, 0x113075a, 0x7, 0x4000018600, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x40000a1a00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 49 [IO wait]: internal/poll.runtime_pollWait(0xffff7006c490, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000190918, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000190900, 0x400256a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000190900, 0x400256a000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011c058, 0x400256a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000304000, 0x400256a000, 0x1000, 0x1000, 0x400010b080, 0x200000003, 0x400010b080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40004046c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40004046c0, 0x400045c00a, 0x400447d8f8, 0x412bc0, 0x40000820c0, 0x400045c000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40004046c0, 0x0, 0x1, 0x0, 0x40000820d8, 0x400045c000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4002523260) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4002523260, 0x40044da000, 0x1000, 0x1000, 0x400447d9b8, 0x44d8a8, 0x80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400018edc0, 0x40044da000, 0x1000, 0x1000, 0x4000117138, 0x4000100a00, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400018edc0, 0x40044da000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400018ee00, 0x40044da000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400447ddd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400447ddd0, 0xa, 0x40044da201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400447ddd0, 0x40044da20a, 0x1, 0x40025379cb, 0x0, 0x40044da263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400447ddd0, 0x400004660a, 0x40025379cb, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000a1a40, 0x4000182660, 0x24, 0x400447df20, 0x4004471ef0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x40000a1a40, 0x113075a, 0x7, 0x4004471ef0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x40000a1a00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 74 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40001b6af0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 79 [IO wait]: internal/poll.runtime_pollWait(0xffff7006c2c0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000190c18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000190c00, 0x40044dd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000190c00, 0x40044dd000, 0x1000, 0x1000, 0x4004479688, 0x69f178, 0x40044796f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010550, 0x40044dd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000304c60, 0x40044dd000, 0x1000, 0x1000, 0x5, 0x4004454500, 0x40044797c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002544b40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002544b40, 0x68f90a, 0x40000ccc60, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4002544b40, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400249eea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400249eea0, 0x4000402000, 0x1000, 0x1000, 0x4004479998, 0x44e884, 0x4004479948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000122940, 0x4000402000, 0x1000, 0x1000, 0x4000117138, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000122940, 0x4000402000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000122980, 0x4000402000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4004479d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004479d80, 0xa, 0x4000402201, 0x1, 0xffff700aed30, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4004479d80, 0x400040220a, 0x1, 0x400018cc40, 0x0, 0x4000402259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4004479d80, 0x400013700a, 0x400018cc40, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000a1a40, 0x400001a480, 0x1f, 0x4004479ee0, 0x4000018930, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x40000a1a40, 0x113075a, 0x7, 0x40004f2fb0, 0xd, 0x4000018930, 0x0, 0x0, 0x2) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x40000a1a00, 0x40004f2fb0, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 77 [select]: net/http.(*persistConn).readLoop(0x4000304120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 78 [select]: net/http.(*persistConn).writeLoop(0x4000304120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 100 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000410c40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 98 [select]: net/http.(*persistConn).readLoop(0x4000304c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 91 [IO wait]: internal/poll.runtime_pollWait(0xffff7006c660, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400012e898, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400012e880, 0x4002509000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400012e880, 0x4002509000, 0x1000, 0x1000, 0x40004eaa48, 0x652ce0, 0x12edac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010908, 0x4002509000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4002540000, 0x4002509000, 0x1000, 0x1000, 0x113674e, 0x412bc0, 0x40003f61e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000083e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000083e00, 0x100a, 0x0, 0x0, 0x40003aabe8, 0x597c30, 0x40003aac38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000083e00, 0x7356e4, 0xffffc2fa994e, 0xe, 0x40003aaca8, 0x40b2f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40045e0600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40045e0600, 0x40045df000, 0x200, 0x200, 0xffffc2fa994e, 0xe, 0x4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400440f840, 0x40045df000, 0x200, 0x200, 0x40004fac07, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400440f840, 0x40045df000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400440f880, 0x40045df000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40045cbce0, 0x680d00, 0x40003aaeb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40045cbce0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40045cbce0, 0xee0d40, 0x40003f6c30, 0x12db068, 0x400440f880) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40045b3950, 0x4002458d20, 0x40045dcb40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 63 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400440f180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 88 [IO wait]: internal/poll.runtime_pollWait(0xffff7006c1d8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000190e18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000190e00, 0x4000361000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000190e00, 0x4000361000, 0x1000, 0x1000, 0x40004edbe0, 0x469b10, 0x40000fb980) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400440c118, 0x4000361000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003050e0, 0x4000361000, 0x1000, 0x1000, 0x40004edcf8, 0x67867c, 0x4004411b60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000082d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000082d80, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40003050e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 89 [select]: net/http.(*persistConn).writeLoop(0x40003050e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 101 [IO wait]: internal/poll.runtime_pollWait(0xffff7006c748, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000101298, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000101280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000101280, 0x58ec48, 0x40044112c0, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004ba2cd8, 0x4714f4, 0x40044112c0, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004ba2cd8, 0x40004f3000, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x400018f700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 102 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x400018fb40, 0x40004ca9a0, 0x40004caa10, 0x4004bac9a0, 0x12e8f90, 0x40046ae2a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x4004b61b84, 0x5, 0x400018f700, 0x12eca70, 0x400018f5c0, 0x130d478, 0x40000e80e0, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40001a27e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40001a27e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 103 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40046ae2a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 104 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40046ae2a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 105 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x40001995f0, 0x400254bf80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 106 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x4000410480) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 107 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x40001aebe0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 108 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x40001aebe0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 109 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x4000010a60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 110 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x40001aec80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 111 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x40001aec80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 112 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x400018fb40, 0x4004bac9a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 117 [IO wait]: internal/poll.runtime_pollWait(0xffff7006c008, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400012ec18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400012ec00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400012ec00, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x400477fbc0, 0xffff993a15b8, 0xffff6962d498, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x400477fbc0, 0x665e1c, 0x0, 0xfe5e60, 0x4004c25dc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x400477fbd8, 0x40047056e0, 0x11ae500, 0x40046ae5c0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40046ae540, 0x12ec710, 0x400477fbd8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40046ae540, 0x12ec710, 0x400477fbd8, 0x40000ea000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 94 [IO wait]: internal/poll.runtime_pollWait(0xffff700cae98, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000101d98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000101d80, 0x4004dc2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000101d80, 0x4004dc2000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010c00, 0x4004dc2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4002541e60, 0x4004dc2000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004dad380) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004dad380, 0xa, 0x4000470123, 0x11320c7, 0x9, 0x597c01, 0x40003aac8f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004dad380, 0xd4fa28, 0x400016e180, 0x46, 0x11320c7, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40045e0f90) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40045e0f90, 0x40045dfe00, 0x200, 0x200, 0x40045bb740, 0x40045bb740, 0x40004eecc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400440fdc0, 0x40045dfe00, 0x200, 0x200, 0x40004fac07, 0xe, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400440fdc0, 0x40045dfe00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400440fe00, 0x40045dfe00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40045e98c0, 0x680d00, 0x40003aaeb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40045e98c0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40045e98c0, 0xee0d40, 0x40003f6f00, 0x12db068, 0x400440fe00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004e32510, 0x4002459200, 0x40045dd680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 143 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40001a2a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 135 [IO wait]: internal/poll.runtime_pollWait(0xffff700caf80, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000101698, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000101680, 0x4004da6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000101680, 0x4004da6000, 0x1000, 0x1000, 0x400242ebe0, 0x469b10, 0x4004bf8180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010af0, 0x4004da6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4002541b00, 0x4004da6000, 0x1000, 0x1000, 0x400242ecf8, 0x67867c, 0x4004be26c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002545e60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4002545e60, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4002541b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 136 [select]: net/http.(*persistConn).writeLoop(0x4002541b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 119 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4004db6140) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 142 [semacquire]: sync.runtime_Semacquire(0x4004c6df38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004c6df30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x40046ab080, 0x4000118a80, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffc2fa9969, 0x29, 0xffffc2fa99cb, 0x25, 0x4004757d40, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004bca870, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffc2fa9969, 0x29, 0xffffc2fa99cb, 0x25, 0x4004757d40, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004bca870, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004da1190, 0x4004da0e60, 0x4000118a80, 0x40001a18c0, 0x4000081e40, 0x0, 0x1bf, 0x4004d9cda8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x40000ea000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004bca870, 0xb, 0x4004bca868, 0x6, 0x4004bca880, 0xd, 0x4004ba16e0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x40000ea000, 0x40003c0000, 0x400009ed50) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x40000ea000, 0x40004fe460, 0x2, 0x2, 0x40004fe470, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 144 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40001a2a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 145 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40001a2a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 146 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40001a2a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 147 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4004db6100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 159 [IO wait]: internal/poll.runtime_pollWait(0xffff700cb068, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000101c98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000101c80, 0x4004dce000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000101c80, 0x4004dce000, 0x1000, 0x1000, 0x4004c26be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010c08, 0x4004dce000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4002541d40, 0x4004dce000, 0x1000, 0x1000, 0x4004c26cf8, 0x67867c, 0x4004be3d40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004dad4a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004dad4a0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4002541d40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 150 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004db42c0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 151 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004db42c0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 152 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4004db42c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 153 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40001a2a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 154 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001843a0, 0x1144559, 0x1b, 0x400034de38, 0x4004be3020, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004bad0a0, 0x4004be3020, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400034df80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004dac6c0, 0x4004bad0a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 160 [select]: net/http.(*persistConn).writeLoop(0x4002541d40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 157 [select]: net/http.(*persistConn).readLoop(0x4002541e60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 158 [select]: net/http.(*persistConn).writeLoop(0x4002541e60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 120 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x4004db4160) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 121 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x4004db4160) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 122 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x4004db4160) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 123 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x4004db4160) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 124 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004c6eb00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 125 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x4004c6eb00, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 126 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x40000ea000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 127 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x40000ea000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 128 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x40000ea000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 129 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:50:35.234+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:50:35.234+00:00 [Info] Setting ipv6=false 2021-08-16T13:50:35.249+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:50:35.251+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:50:35.251+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:50:35.251+00:00 [Info] Setting log level to Info 2021-08-16T13:50:35.251+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:50:35.254+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:50:35 audit: created new audit service 2021-08-16T13:50:35.264+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:50:35.264+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:50:35.264+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:50:35.264+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:50:35.264+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:50:35.275+00:00 [Info] security context initialized 2021-08-16T13:50:35.275+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:50:35.275+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:50:35.275+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:50:35.275+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:50:35.275+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:50:35.288+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:50:35.290+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:50:35.290+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:50:35.313+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:50:35.315+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:50:35.315+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:50:35.315+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:50:35.317+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:50:35.317+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:50:35.318+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:50:35.321+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:50:35.369+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:50:35.369+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 59 us 2021-08-16T13:50:35.369+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:50:35.372+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:50:35.373+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:50:35.375+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:50:35.375+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:50:35.375+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:50:35.375+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:50:35.375+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:50:35.376+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:35.376+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:35.378+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:50:35.378+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:50:35.378+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:50:35.378+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:35.379+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:50:35.379+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:50:35.379+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:50:35.379+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:50:35.379+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:50:35.379+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:50:35.379+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:50:35.384+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 4.696409ms 2021-08-16T13:50:35.384+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:50:35.385+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:50:35.385+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:35.385+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:50:35.386+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:50:35.386+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:50:35.386+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:50:35.395+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:50:35.395+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:50:35.395+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:50:35.396+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:50:35.396+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 2021-08-16T13:50:35.396+00:00 [Info] RebalanceMgr::initService Init 2021-08-16T13:50:35.396+00:00 [Info] ServiceMgr::registerWithServer nodeuuid 8a95cde683482bdec453edbb8b18eb5d 0x4004edc01f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x4004edc01f pc=0xbd8ea4] goroutine 178 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004dc69f0 sp=0x4004dc69c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004dc6a30 sp=0x4004dc69f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x40043bf860, 0x4004cc4c00, 0x11a9890, 0x4004dc6b40, 0x40043bf878, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004dc6ab0 sp=0x4004dc6a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x40043bf860, 0x4004cc4c00, 0x11a9890, 0x4004dc6b40, 0x40043bf878, 0xc18f8c, 0x4004dc6c18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004dc6af0 sp=0x4004dc6ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x40024268c0, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004dc6da0 sp=0x4004dc6af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x40024268c0, 0x40004af7a0, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004dc6e60 sp=0x4004dc6da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x400472ddc0, 0x400472ddd0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004dc7200 sp=0x4004dc6e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x400472ddc0, 0x400472ddd0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004dc78f0 sp=0x4004dc7200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004e9b3e0, 0x4004c85880, 0x4004673080, 0x400472dda0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004dc7fb0 sp=0x4004dc78f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004dc7fb0 sp=0x4004dc7fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x4000182000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x400008d680, 0x4, 0x1136a87, 0xf, 0x40001a1e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 5 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400000e120) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 6 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400000e138) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 7 [IO wait]: internal/poll.runtime_pollWait(0xffff6d3b9758, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000190118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000190100, 0x40001a9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000190100, 0x40001a9000, 0x1000, 0x1000, 0x1311860, 0x431a7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400013e4f0, 0x40001a9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x40001404e0, 0x400046a001, 0xdff, 0xdff, 0x40001238e8, 0x6902d8, 0x400019d948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x40001305a0, 0x400046a001, 0xdff, 0xdff, 0x400019d9f8, 0x6f1968, 0x400019d938) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x4000222000, 0xa, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4000222000, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4000222000, 0xf5a960, 0x4000200260, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4000200240, 0x40000466e0, 0x400019db28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x40002100a0, 0x12f8f98, 0x4000200240, 0x40000372f7, 0x16, 0x400046c120, 0x4000123c01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x40002100a0, 0x12f8f98, 0x4000200240, 0x40002100a0, 0x400020c078, 0x400020c080, 0x4000220000, 0x400020a020, 0xee0980, 0x400046c120, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x40002100a0, 0x12f8f98, 0x4000200240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x400008c600, 0x4000123f78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400011ff78, 0x400008c600, 0x12db8a8, 0x4000130570, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x400008c600, 0x400009c460, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x400008c600, 0x400009c460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 8 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 9 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 10 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 11 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 12 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 20 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400010c500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 13 [syscall]: syscall.Syscall(0x3f, 0x0, 0x400042bfa0, 0x4, 0x400042be68, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x400042bfa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000140000, 0x400042bfa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x400013e000, 0x400042bfa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 38 [select]: net/http.(*persistConn).writeLoop(0x400014ea20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 26 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400006c750, 0x12c18bc, 0x1, 0x4000121e58, 0x40000381e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x4000428ec8, 0x40000381e0, 0x4000428ec8, 0x679704) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000121fa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x400020e000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 24 [select]: net/http.(*persistConn).readLoop(0x400014e480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 25 [select]: net/http.(*persistConn).writeLoop(0x400014e480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 27 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x400020e000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 39 [IO wait]: internal/poll.runtime_pollWait(0xffff6d3b9670, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000220198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000220180, 0x40001a8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000220180, 0x40001a8000, 0x1000, 0x1000, 0x4000065a88, 0x68f7a4, 0x40000985d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400013e0d0, 0x40001a8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400014e480, 0x40001a8000, 0x1000, 0x1000, 0xf12180, 0x400014a040, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000200600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000200600, 0xa, 0x1008880, 0x400014a000, 0x199, 0x1000, 0x4000065b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000200600, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400008c810) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400008c810, 0x4000226000, 0xe00, 0xe00, 0x4000065da8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000160580, 0x4000226000, 0xe00, 0xe00, 0x4000065e08, 0x40bb94, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000160580, 0x4000226000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000161580, 0x4000226000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4000098580, 0x0, 0x4000065eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4000098580, 0x0, 0x0, 0x4000065f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4000098580, 0xee0d40, 0x400014a000, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40004ac000, 0x4000140960, 0x40004a2120) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 37 [IO wait]: internal/poll.runtime_pollWait(0xffff6d3b9588, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000220298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000220280, 0x40004a0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000220280, 0x40004a0000, 0x1000, 0x1000, 0x400048ebe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010008, 0x40004a0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400014ea20, 0x40004a0000, 0x1000, 0x1000, 0x400048ecf8, 0x67867c, 0x40004a20c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000140780) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000140780, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400014ea20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 48 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400020e340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 49 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x40003e2ea0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 47 [select]: net/http.(*persistConn).writeLoop(0x400046cc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 32 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000080680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 46 [select]: net/http.(*persistConn).readLoop(0x400046cc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 40 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x40001887e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 56 [IO wait]: internal/poll.runtime_pollWait(0xffff6d3b8c78, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400008ef98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400008ef80, 0x40003cb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400008ef80, 0x40003cb000, 0x1000, 0x1000, 0x40001e3688, 0x69f178, 0x40001e36f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40004860b0, 0x40003cb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004398480, 0x40003cb000, 0x1000, 0x1000, 0x5, 0x4004480000, 0x40001e37c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000201620) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000201620, 0x68f90a, 0x40000d9130, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000201620, 0x0, 0x40001e3868, 0x41a0a8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40004c3a40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40004c3a40, 0x400241d000, 0x1000, 0x1000, 0x40001e3998, 0x44e884, 0x1c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400020e540, 0x400241d000, 0x1000, 0x1000, 0x40004806b8, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400020e540, 0x400241d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400020e580, 0x400241d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40001e3d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001e3d80, 0xa, 0x400241d201, 0x1, 0xffff6c977700, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40001e3d80, 0x400241d20a, 0x1, 0x400001a8c0, 0x0, 0x400241d259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40001e3d80, 0x400009940a, 0x400001a8c0, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400011d700, 0x40003c63a0, 0x1f, 0x40001e3ee0, 0x40003c4720, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x400011d700, 0x113075a, 0x7, 0x4004404ee0, 0xd, 0x40003c4720, 0x0, 0x0, 0x2) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x400011d6c0, 0x4004404ee0, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 75 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400008aeb0, 0xd00007, 0x40047ce884, 0x5, 0x40046fcee0, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 43 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x40001887e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 44 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x40001887e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 45 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x40001887e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 74 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 66 [IO wait]: internal/poll.runtime_pollWait(0xffff6d3b9018, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000190918, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000190900, 0x40001ea000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000190900, 0x40001ea000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000206120, 0x40001ea000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000cc60, 0x40001ea000, 0x1000, 0x1000, 0x400042ec00, 0x200000003, 0x400042ec00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000480c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000480c60, 0x400023200a, 0x40001238c8, 0x4d3b10, 0x4000140060, 0x4000232000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000480c60, 0x0, 0x1, 0x0, 0x4000140078, 0x4000232000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40001b0420) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40001b0420, 0x40003c9000, 0x1000, 0x1000, 0x40001239b8, 0x0, 0x4000123988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000161a80, 0x40003c9000, 0x1000, 0x1000, 0x4000103e00, 0x3, 0x4000123be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000161a80, 0x40003c9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000161ac0, 0x40003c9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000123dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000123dd0, 0xa, 0x40003c9201, 0x1, 0xd1f, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000123dd0, 0x40003c920a, 0x1, 0x4004404ec9, 0x0, 0x40003c92e1, 0x1, 0xd1f, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000123dd0, 0x400441a40a, 0x4004404ec9, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400011d700, 0x400016e8e8, 0x17, 0x4000123f20, 0x40024ab6b0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x400011d700, 0x113075a, 0x7, 0x40024ab6b0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x400011d6c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 67 [IO wait]: internal/poll.runtime_pollWait(0xffff6d3b8e48, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000190698, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000190680, 0x40024ba000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000190680, 0x40024ba000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400013e618, 0x40024ba000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400046cc60, 0x40024ba000, 0x1000, 0x1000, 0x400042ed80, 0x200000003, 0x400042ed80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003e3da0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003e3da0, 0x400023200a, 0x40001e98c8, 0x4d3b10, 0x4000140060, 0x4000232000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003e3da0, 0x0, 0x1, 0x0, 0x4000140078, 0x4000232000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40001b03c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40001b03c0, 0x40003c8000, 0x1000, 0x1000, 0x40001e99b8, 0x44d8a8, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000161a00, 0x40003c8000, 0x1000, 0x1000, 0x4000141198, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000161a00, 0x40003c8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000161a40, 0x40003c8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40001e9dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001e9dd0, 0xa, 0x40003c8201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40001e9dd0, 0x40003c820a, 0x1, 0x40024a3feb, 0x0, 0x40003c8263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40001e9dd0, 0x400020a20a, 0x40024a3feb, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400011d700, 0x400009e6f0, 0x24, 0x40001e9f20, 0x40024ab698, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x400011d700, 0x113075a, 0x7, 0x40024ab698, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x400011d6c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 76 [IO wait]: internal/poll.runtime_pollWait(0xffff6d3b8aa8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000191718, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000191700, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000191700, 0x0, 0x400438ad88, 0x665fe4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40047e0450, 0x611a6d2b, 0x400438adb8, 0x445c24) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40047e0450, 0x400438ae20, 0x400438ae28, 0x18, 0x400042ef00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x400441ce00, 0x12ed9d0, 0x40047e0450, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x400441ce00, 0x12ed9d0, 0x40047e0450, 0x4000182000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 70 [select]: net/http.(*persistConn).readLoop(0x400000cc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 71 [select]: net/http.(*persistConn).writeLoop(0x400000cc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 117 [IO wait]: internal/poll.runtime_pollWait(0xffff6d3b8d60, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400008ee18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400008ee00, 0x40003cf000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400008ee00, 0x40003cf000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40004860b8, 0x40003cf000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004398360, 0x40003cf000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40002016e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40002016e0, 0x100a, 0x0, 0x0, 0x40000533b8, 0x456e60, 0x4000053438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40002016e0, 0x0, 0x4000053478, 0x655784, 0x40000534a8, 0x40b2f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40001b14a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40001b14a0, 0x4004403a00, 0x200, 0x200, 0x4, 0x0, 0x4000053508) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000468140, 0x4004403a00, 0x200, 0x200, 0x40002087e7, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000468140, 0x4004403a00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000468180, 0x4004403a00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40002414a0, 0x680d00, 0x40000536b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40002414a0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40002414a0, 0xee0d40, 0x400014a690, 0x12db068, 0x4000468180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40004ad5f0, 0x40044bc120, 0x4004419a40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 63 [select]: net/http.(*persistConn).readLoop(0x4004398360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 64 [select]: net/http.(*persistConn).writeLoop(0x4004398360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 61 [select]: net/http.(*persistConn).readLoop(0x4004398480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 62 [select]: net/http.(*persistConn).writeLoop(0x4004398480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 82 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x40004c23c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 83 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x40043c2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 84 [IO wait]: internal/poll.runtime_pollWait(0xffff6d3b8b90, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40043c2118, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40043c2100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40043c2100, 0x680da0, 0x4004395e98, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40043a8288, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40043a8288, 0x12ed9d0, 0x40043a8288, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x40043c2080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 85 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4000095040, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 86 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4000095040, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 87 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x4000095040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 88 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x40004c33b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 149 [select]: net/http.(*persistConn).writeLoop(0x400473c900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 73 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400006c750, 0x1144559, 0x1b, 0x40001e5e38, 0x400457d920, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x40046fced0, 0x400457d920, 0x4004395e98, 0xccadf0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x40004c3830) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40001e5f80, 0x40043a8288, 0x1000000000001) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40047ae960, 0x40046fced0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 91 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x4000180150) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 72 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x4000143ea0, 0x400457d800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 100 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000161e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 97 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x4004c5e100, 0x40004aeee0, 0x40004aef50, 0x40045c1300, 0x12e8f90, 0x4002422540, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x4004c80164, 0x5, 0x400020fb80, 0x12eca70, 0x400020fa40, 0x130d478, 0x4000180230, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40024228c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40024228c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 96 [IO wait]: internal/poll.runtime_pollWait(0xffff6d3b89c0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40043c2e18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40043c2e00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40043c2e00, 0x0, 0x4000000004, 0x94fb58) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40047779c8, 0x4714f4, 0x1be18c0, 0x4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40047779c8, 0x0, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x400020fb80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 95 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400020f380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 94 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000081680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 105 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x4000143e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 124 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40024fa540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 123 [semacquire]: sync.runtime_Semacquire(0x4004e9b3e8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004e9b3e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x4004673080, 0x400011be60, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffd4941969, 0x29, 0xffffd49419cb, 0x25, 0x4004c86fc0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004c80e50, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffd4941969, 0x29, 0xffffd49419cb, 0x25, 0x4004c86fc0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004c80e50, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004dc7190, 0x4004dc6e60, 0x400011be60, 0x40004b6900, 0x4000080680, 0x0, 0xff, 0x4002414da8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x4000182000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004c80e50, 0xb, 0x4004c80e48, 0x6, 0x4004c80e60, 0xd, 0x40047795a0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x4000182000, 0x40003e0000, 0x40002021e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x4000182000, 0x400006de40, 0x2, 0x2, 0x400006de50, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 106 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x4000143e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 107 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x4004c5e100, 0x40045c1300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 108 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x4002422540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 109 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x4002422540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 110 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x40004ac630, 0x40045f2ba0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 111 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x4000161dc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 112 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x4000143ea0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 113 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x4000143ea0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 130 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x400013e8a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 77 [IO wait]: internal/poll.runtime_pollWait(0xffff6d3b88d8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000191818, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000191800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000191800, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40047e0468, 0xffff9665c5b8, 0xffff6c430580, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40047e0468, 0x665e1c, 0x0, 0xfe5e60, 0x400042c5c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x40047e0480, 0x4004785200, 0x11ae500, 0x400441cf60, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x400441cee0, 0x12ec710, 0x40047e0480, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x400441cee0, 0x12ec710, 0x40047e0480, 0x4000182000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 151 [select]: net/http.(*persistConn).roundTrip(0x400473c900, 0x4000468b80, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004f0c300, 0xf8, 0x10a8660, 0x4000418e18) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004f0c300, 0x1b6e900, 0xc03eb948d79f7fd7, 0x1bfa887842) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004f0c200, 0x12db048, 0x1b6e900, 0xc03eb948d79f7fd7, 0x1bfa887842, 0x1be4a80, 0x0, 0x4000486278, 0x40044d2bd0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x40044d2ba0, 0x4004f0c200, 0xc03eb948d79f7fd7, 0x1bfa887842, 0x1be4a80, 0x4000486278, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x40044d2ba0, 0x4004f0c200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x40002097a0, 0x1b, 0x4000419580, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x40044d0b40, 0x112f905, 0x6, 0x12db908, 0x4000046ca0, 0xee40c0, 0x400008fa18, 0x4000419580, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x400003e580, 0x46, 0x12db908, 0x4000046ca0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x400003e580, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x20, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000419d40, 0x4, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x40000d5680, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x40000d5680, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000468740) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1267 +0xf8 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 137 [IO wait]: internal/poll.runtime_pollWait(0xffff6d3b8f30, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400008f898, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400008f880, 0x40044dc000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400008f880, 0x40044dc000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000486218, 0x40044dc000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004398c60, 0x40044dc000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40044bd0e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40044bd0e0, 0xa, 0x4004d8f3e8, 0x48731c, 0x9, 0x1, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40044bd0e0, 0x0, 0x4004d8f448, 0x57d438, 0x11320c7, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004e0e570) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004e0e570, 0x4004befc00, 0x200, 0x200, 0x4004c3f320, 0x4004c3f320, 0x4000043320) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004c5e9c0, 0x4004befc00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004c5e9c0, 0x4004befc00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004c5ea00, 0x4004befc00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004e0ba20, 0x0, 0x4004d8f6b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004e0ba20, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004e0ba20, 0xee0d40, 0x40002114a0, 0x12db068, 0x4004c5ea00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004d79440, 0x40045bfce0, 0x40045f3d40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 119 [IO wait]: internal/poll.runtime_pollWait(0xffff6c6ec090, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400008f398, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400008f380, 0x400045f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400008f380, 0x400045f000, 0x1000, 0x1000, 0x4004462be0, 0x469b10, 0x4004459800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000486138, 0x400045f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004c7ca20, 0x400045f000, 0x1000, 0x1000, 0x4004462cf8, 0x67867c, 0x4004419c20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40044bc180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40044bc180, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004c7ca20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 120 [select]: net/http.(*persistConn).writeLoop(0x4004c7ca20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 148 [IO wait]: internal/poll.runtime_pollWait(0xffff6c6ec178, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40043c2f18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40043c2f00, 0x4004cb4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40043c2f00, 0x4004cb4000, 0x1000, 0x1000, 0x40004d9be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000105d8, 0x4004cb4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400473c900, 0x4004cb4000, 0x1000, 0x1000, 0x40004d9cf8, 0x67867c, 0x4004c865a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004cb2120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004cb2120, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400473c900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 125 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40024fa540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 126 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40024fa540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 127 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40024fa540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 128 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4000468700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 129 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4000239550, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 162 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4000239550, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 163 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4000239550) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 164 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40024fa540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 165 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400006c750, 0x1144559, 0x1b, 0x4000417e38, 0x40044ca420, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x40001b3700, 0x40044ca420, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000417f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40044bc660, 0x40001b3700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 170 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).processSchedIndexes(0x40002393f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:296 +0x44 created by github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:218 +0x3c goroutine 168 [select]: net/http.(*persistConn).readLoop(0x4004398c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 169 [select]: net/http.(*persistConn).writeLoop(0x4004398c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 152 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x40002393f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:224 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 153 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x40002393f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:612 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 154 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x40002393f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:662 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 155 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x40002393f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:752 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 156 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x400477b000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:249 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 195 [select]: net/http.(*persistConn).roundTrip(0x4004399320, 0x4000468e80, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004f0cc00, 0xf8, 0x10a8660, 0x4004dba588) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004f0cc00, 0x1b6e900, 0xc03eb948d7a677de, 0x1bfa8f7049) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004f0cb00, 0x12db048, 0x1b6e900, 0xc03eb948d7a677de, 0x1bfa8f7049, 0x1be4a80, 0x0, 0x40004862d0, 0x40044d3590, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x40044d3560, 0x4004f0cb00, 0xc03eb948d7a677de, 0x1bfa8f7049, 0x1be4a80, 0x40004862d0, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x40044d3560, 0x4004f0cb00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4000209c80, 0x1b, 0x4004dbacf0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x40044d1710, 0x112f905, 0x6, 0x12db908, 0x4000046d80, 0xee40c0, 0x400008ff18, 0x4004dbacf0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x400003e700, 0x46, 0x12db908, 0x4000046d80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x400003e700, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x40044d34a0, 0xffff6d3bbe38) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004dcb4b0, 0x1063520, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x4004f1a240, 0x4004f1a240, 0x400003e700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.FetchNewClusterInfoCache(0xffffd494194e, 0xe, 0x113075a, 0x7, 0x1130e44, 0x8, 0x4004f2c0c4, 0x2, 0xf01) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:123 +0x134 github.com/couchbase/indexing/secondary/manager.(*IndexManager).FetchNewClusterInfoCache(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:306 github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).getIndexMetadata(0x1be5520, 0x12ec7a0, 0x4000468e40, 0x400014af00, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:1284 +0x74 github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).handleIndexMetadataRequest(0x1be5520, 0x12edbe0, 0x40024fa700, 0x4004f0ca00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:1271 +0x3e8 net/http.HandlerFunc.ServeHTTP(0x40046fcdd0, 0x12edbe0, 0x40024fa700, 0x4004f0ca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2069 +0x40 net/http.(*ServeMux).ServeHTTP(0x4000171dc0, 0x12edbe0, 0x40024fa700, 0x4004f0ca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2448 +0x190 net/http.serverHandler.ServeHTTP(0x400441ce00, 0x12edbe0, 0x40024fa700, 0x4004f0ca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2887 +0xbc net/http.(*conn).serve(0x400009dd60, 0x12f86a0, 0x4000468cc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:1952 +0x71c created by net/http.(*Server).Serve /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:3013 +0x318 goroutine 158 [select]: net/http.(*persistConn).roundTrip(0x4004399200, 0x4000468c00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004f0c800, 0xf8, 0x10a8660, 0x4004dbcdd8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004f0c800, 0x1b6e900, 0xc03eb948d7a265aa, 0x1bfa8b5e15) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004f0c700, 0x12db048, 0x1b6e900, 0xc03eb948d7a265aa, 0x1bfa8b5e15, 0x1be4a80, 0x0, 0x40004862a0, 0x40044d2f00, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x40044d2ed0, 0x4004f0c700, 0xc03eb948d7a265aa, 0x1bfa8b5e15, 0x1be4a80, 0x40004862a0, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x40044d2ed0, 0x4004f0c700, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x40002099b0, 0x1b, 0x4004dbd540, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x40044d1050, 0x112f905, 0x6, 0x12db908, 0x4000046d20, 0xee40c0, 0x400008fb98, 0x4004dbd540, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x400003e680, 0x46, 0x12db908, 0x4000046d20, 0x12f8668, 0x4004f0c600, 0x40001b3f20, 0x40002098c0, 0xffff6d3afa78, 0x4000468bc0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x400003e680, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x240, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004dbdd00, 0x40044d0240, 0xffff6d3bbe38) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x4004f1a000, 0xd47600, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x4004f1a000, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x4000182000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9702 +0x344 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 159 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x4000182000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1090 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 160 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x4000182000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1017 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 161 [sleep]: time.Sleep(0x12a05f200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/plasma.RunMemQuotaTunerIteration(0x40004c9fb0, 0xdf74d8, 0x400010e600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:211 +0x24c github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:247 +0xc0 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 goroutine 171 [runnable]: internal/poll.runtime_pollWait(0xffff6c6ebdd8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400008fd98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400008fd80, 0x4004f29000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400008fd80, 0x4004f29000, 0x1000, 0x1000, 0x0, 0x4004f28000, 0x4004db9a38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40004862b0, 0x4004f29000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).fill(0x4004f261e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004f261e0, 0x654f0a, 0x4004db9b08, 0x439548, 0x4004db9af8, 0x5429e4, 0x400008fd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).ReadLine(0x4004f261e0, 0x655058, 0x4004db9b08, 0x655070, 0x4004db9bc8, 0x412ff8, 0x40044d14d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:389 +0x30 net/textproto.(*Reader).readLineSlice(0x4004db9cf0, 0x40044d14d0, 0x4004db9c48, 0x6567e4, 0x452b97f5b0d064d4, 0xf26100) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/textproto/reader.go:57 +0xac net/textproto.(*Reader).ReadLine(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/textproto/reader.go:38 net/http.ReadResponse(0x4004f261e0, 0x4004f0c900, 0x1000, 0x4004f29000, 0x40004862b0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/response.go:161 +0x8c github.com/couchbase/cbauth/revrpc.(*Service).Run(0x40044d3080, 0x4004db9f10, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:132 +0x2ac github.com/couchbase/cbauth/revrpc.BabysitService(0x4004db9f10, 0x40044d3080, 0x0, 0x0, 0x0, 0xd8d03c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth/service.RegisterManager(0x13007d8, 0x400477b000, 0x0, 0x0, 0x4004c61f80, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:146 +0xa0 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).registerWithServer(0x400477b000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:301 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:205 +0x78 goroutine 172 [select]: net/http.(*persistConn).roundTrip(0x4004398fc0, 0x4000468bc0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004f0c600, 0xf8, 0x10a8660, 0x4004c61798) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004f0c600, 0x1b6e900, 0xc03eba56d7a0ff45, 0x1176f8e27b0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004f0c500, 0x12db048, 0x1b6e900, 0xc03eba56d7a0ff45, 0x1176f8e27b0, 0x1be4a80, 0x0, 0x4000486288, 0x40044d2cc0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x40044d2c90, 0x4004f0c500, 0xc03eba56d7a0ff45, 0x1176f8e27b0, 0x1be4a80, 0x4000486288, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x40044d2c90, 0x4004f0c500, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004f07120, 0x1f, 0x4004c61d10, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/indexer.getWithAuth(0x4004f07120, 0x1f, 0xe, 0x113846e, 0x11) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:3077 +0x68 github.com/couchbase/indexing/secondary/indexer.getGlobalTopology(0x4004e9aa00, 0xe, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:2266 +0x78 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).updateNodeList(0x400477b000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:220 +0x3c created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:206 +0x98 goroutine 177 [IO wait]: internal/poll.runtime_pollWait(0xffff6c6ebfa8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400008fb18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400008fb00, 0x4004f2a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400008fb00, 0x4004f2a000, 0x1000, 0x1000, 0x40043923e8, 0xffff6d3bba80, 0x88) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40004862b8, 0x4004f2a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004398fc0, 0x4004f2a000, 0x1000, 0x1000, 0x60, 0x0, 0x4004f07101) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004f262a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004f262a0, 0x1, 0x4004f2e000, 0x43d584, 0x0, 0xd8c91c, 0x4004392568) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004398fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 78 [runnable]: syscall.Syscall(0x3f, 0x1d, 0x40047ec000, 0x1000, 0x502, 0x40047ec000, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x1d, 0x40047ec000, 0x1000, 0x1000, 0x0, 0x12dbfe8, 0x1b7f708) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x400008fc80, 0x40047ec000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 net.(*netFD).Read(0x400008fc80, 0x40047ec000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000206428, 0x40047ec000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004399200, 0x40047ec000, 0x1000, 0x1000, 0x60, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40047aef60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40047aef60, 0x1, 0x400457db60, 0x43d584, 0x0, 0x0, 0x4004ec3d68) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004399200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 194 [select]: net/http.(*persistConn).writeLoop(0x4004398fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 79 [select]: net/http.(*persistConn).writeLoop(0x4004399200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 196 [IO wait]: internal/poll.runtime_pollWait(0xffff6c6ebcf0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400008fe18, 0x72, 0x0, 0x1, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400008fe00, 0x40044d3331, 0x1, 0x1, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400008fe00, 0x40044d3331, 0x1, 0x1, 0x4004ec3701, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40004862c0, 0x40044d3331, 0x1, 0x1, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*connReader).backgroundRead(0x40044d3320) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:692 +0x50 created by net/http.(*connReader).startBackgroundRead /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:688 +0xc4 goroutine 199 [IO wait]: internal/poll.runtime_pollWait(0xffff6c6ebc08, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004f34018, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004f34000, 0x4004f36000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004f34000, 0x4004f36000, 0x1000, 0x1000, 0x40043923e8, 0xffff6d3baa08, 0x88) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40004862e0, 0x4004f36000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004399320, 0x4004f36000, 0x1000, 0x1000, 0x60, 0x0, 0x4004f07101) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004f26840) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004f26840, 0x1, 0x4004f2e420, 0x43d584, 0x0, 0xd8c91c, 0x4004392568) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004399320) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 200 [select]: net/http.(*persistConn).writeLoop(0x4004399320) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc Initializing write barrier = 8000 2021-08-16T13:50:40.549+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:50:40.549+00:00 [Info] Setting ipv6=false 2021-08-16T13:50:40.554+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:50:40.556+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:50:40.556+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:50:40.556+00:00 [Info] Setting log level to Info 2021-08-16T13:50:40.557+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:50:40.559+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:50:40 audit: created new audit service 2021-08-16T13:50:40.568+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:50:40.568+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:50:40.569+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:50:40.569+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:50:40.569+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:50:40.579+00:00 [Info] security context initialized 2021-08-16T13:50:40.579+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:50:40.579+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:50:40.579+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:50:40.579+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:50:40.579+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:50:40.592+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:50:40.606+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:50:40.612+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:50:40.624+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:50:40.626+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:50:40.626+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:50:40.627+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:50:40.631+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:50:40.632+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:50:40.633+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:50:40.634+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:50:40.679+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:50:40.680+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 54 us 2021-08-16T13:50:40.680+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:50:40.682+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:50:40.682+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:50:40.686+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:50:40.686+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:50:40.686+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:50:40.686+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:50:40.686+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:50:40.686+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:40.686+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:40.688+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:50:40.688+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:50:40.688+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:50:40.688+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:40.688+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:50:40.688+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:50:40.688+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:50:40.688+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:50:40.688+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:50:40.688+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:50:40.688+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:50:40.693+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 4.375097ms 2021-08-16T13:50:40.693+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:50:40.694+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:50:40.694+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:40.694+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:50:40.695+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:50:40.695+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:50:40.695+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... unexpected fault address 0x4004c98d9f2021-08-16T13:50:40.709+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:50:40.709+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:50:40.709+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:50:40.709+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:50:40.709+00:00 [Info] : Recover shard shard1 from metadata completed. fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x4004c98d9f pc=0xbd8ea4] goroutine 186 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x400145e9f0 sp=0x400145e9c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x400145ea30 sp=0x400145e9f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x400147dba0, 0x40047ef800, 0x11a9890, 0x400145eb40, 0x400147dbb8, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x400145eab0 sp=0x400145ea40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x400147dba0, 0x40047ef800, 0x11a9890, 0x400145eb40, 0x400147dbb8, 0xc18f8c, 0x400145ec18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x400145eaf0 sp=0x400145eab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x40024a0b40, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x400145eda0 sp=0x400145eaf0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x40024a0b40, 0x400141b9d0, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x400145ee60 sp=0x400145eda0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004c59d80, 0x4004c59d90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x400145f200 sp=0x400145ee60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004c59d80, 0x4004c59d90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x400145f8f0 sp=0x400145f200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004e572c0, 0x4004e5ca80, 0x4004653600, 0x4004c59d60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x400145ffb0 sp=0x400145f8f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x400145ffb0 sp=0x400145ffb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x4001426000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x40000ff0e0, 0x4, 0x1136a87, 0xf, 0x40000f9e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 7 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400000e678) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 8 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400000e690) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 9 [IO wait]: internal/poll.runtime_pollWait(0xffff8052ae88, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001a4118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40001a4100, 0x40001bb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40001a4100, 0x40001bb000, 0x1000, 0x1000, 0x1311860, 0x47b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000186018, 0x40001bb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x40001801e0, 0x4000122001, 0xdff, 0xdff, 0x400009b8e8, 0x6902d8, 0x40000f5948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x40001a2048, 0x4000122001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40001b8160, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40001b8160, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40001b8160, 0xf5a960, 0x40001802c0, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x40001802a0, 0x4000116100, 0x40000f5b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x40001c4000, 0x12f8f98, 0x40001802a0, 0x40001187f7, 0x16, 0x4000126120, 0x400009bc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x40001c4000, 0x12f8f98, 0x40001802a0, 0x40001c4000, 0x400018c0e0, 0x400018c0f0, 0x40001a4180, 0x400018a080, 0xee0980, 0x4000126120, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x40001c4000, 0x12f8f98, 0x40001802a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x40000fe060, 0x400009bf78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x40000a1f78, 0x40000fe060, 0x12db8a8, 0x40001a2018, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x40000fe060, 0x40000b1360, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x40000fe060, 0x40000b1360) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 19 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400010c100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 10 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 11 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 12 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 13 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 14 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 15 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40003f27a0, 0x4, 0x40003f2668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40003f27a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x40003f27a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x40003f27a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 99 [select]: net/http.(*persistConn).writeLoop(0x4000126ea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 23 [select]: net/http.(*persistConn).readLoop(0x40003d46c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 24 [select]: net/http.(*persistConn).writeLoop(0x40003d46c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 36 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400021c310, 0x12c18bc, 0x1, 0x400009de58, 0x400016a000, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x40003f36c8, 0x400016a000, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400009dfa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4000114180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 37 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4000114180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 48 [select]: net/http.(*persistConn).readLoop(0x40003d4c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 40 [IO wait]: internal/poll.runtime_pollWait(0xffff8052acb8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001a4218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40001a4200, 0x4001450000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40001a4200, 0x4001450000, 0x1000, 0x1000, 0x4000151be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000186088, 0x4001450000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001ae5a0, 0x4001450000, 0x1000, 0x1000, 0x4000151cf8, 0x67867c, 0x4001422a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000180660) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000180660, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40001ae5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 41 [select]: net/http.(*persistConn).writeLoop(0x40001ae5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 50 [IO wait]: internal/poll.runtime_pollWait(0xffff8052ada0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0100, 0x40001ba000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0100, 0x40001ba000, 0x1000, 0x1000, 0x400014ca88, 0x68f7a4, 0x400043e050) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000100d0, 0x40001ba000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003d46c0, 0x40001ba000, 0x1000, 0x1000, 0xf12180, 0x4000158220, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003a93e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003a93e0, 0xa, 0x1008880, 0x40001581e0, 0x199, 0x1000, 0x400014cb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003a93e0, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40001100c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40001100c0, 0x4001452000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000114000, 0x4001452000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000114000, 0x4001452000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000114080, 0x4001452000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x400043e000, 0x0, 0x400014ceb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400043e000, 0x0, 0x0, 0x400014cf70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400043e000, 0xee0d40, 0x40001581e0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400042e000, 0x40014200c0, 0x400016a060) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 44 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x400046a540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 31 [IO wait]: internal/poll.runtime_pollWait(0xffff8052a660, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0518, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0500, 0x4002489000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0500, 0x4002489000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000186250, 0x4002489000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003d4c60, 0x4002489000, 0x1000, 0x1000, 0x4001414600, 0x200000003, 0x4001414600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000181c20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000181c20, 0x400044800a, 0x400009f8c8, 0x4d3b10, 0x40000820c0, 0x4000448000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000181c20, 0x0, 0x1, 0x0, 0x40000820d8, 0x4000448000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4001424600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4001424600, 0x400141e000, 0x1000, 0x1000, 0x400009f9b8, 0x0, 0x400009f988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000080000, 0x400141e000, 0x1000, 0x1000, 0x4001414780, 0x3, 0x400009fbe0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000080000, 0x400141e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000080140, 0x400141e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400009fdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400009fdd0, 0xa, 0x400141e201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400009fdd0, 0x400141e20a, 0x1, 0x400041f089, 0x0, 0x400141e2e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400009fdd0, 0x400012d10a, 0x400041f089, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400012d080, 0x400003a9c0, 0x17, 0x400009ff20, 0x400038e450, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x400012d080, 0x113075a, 0x7, 0x400038e450, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x400012d040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 53 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400018e7c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 28 [select]: net/http.(*persistConn).writeLoop(0x40003d47e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 27 [select]: net/http.(*persistConn).readLoop(0x40003d47e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 45 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x400046a540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 46 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x400046a540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 47 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x400046a540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 49 [select]: net/http.(*persistConn).writeLoop(0x40003d4c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 32 [IO wait]: internal/poll.runtime_pollWait(0xffff8052a578, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0318, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0300, 0x400017b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0300, 0x400017b000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000102f8, 0x400017b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003d47e0, 0x400017b000, 0x1000, 0x1000, 0x40003e3080, 0x200000003, 0x40003e3080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003a9b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003a9b00, 0x400044800a, 0x400009b8c8, 0x4d3b10, 0x40000820c0, 0x4000448000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003a9b00, 0x0, 0x1, 0x0, 0x40000820d8, 0x4000448000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4001424660) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4001424660, 0x400141f000, 0x1000, 0x1000, 0x400009b9b8, 0x44d8a8, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40000805c0, 0x400141f000, 0x1000, 0x1000, 0x4000180bf8, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40000805c0, 0x400141f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40000815c0, 0x400141f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400009bdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400009bdd0, 0xa, 0x400141f201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400009bdd0, 0x400141f20a, 0x1, 0x400047a0db, 0x0, 0x400141f263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400009bdd0, 0x400011660a, 0x400047a0db, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400012d080, 0x4000042c60, 0x24, 0x400009bf20, 0x400038e438, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x400012d080, 0x113075a, 0x7, 0x400038e438, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x400012d040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 57 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400018ebc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 76 [select]: net/http.(*persistConn).readLoop(0x40003d5200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 118 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40003d73c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 33 [IO wait]: internal/poll.runtime_pollWait(0xffff8052a3a8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0d18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0d00, 0x40024e1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0d00, 0x40024e1000, 0x1000, 0x1000, 0x40001cf688, 0x69f178, 0x40001cf6f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000186068, 0x40024e1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000126ea0, 0x40024e1000, 0x1000, 0x1000, 0x5, 0x40043cea00, 0x40001cf7c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400010e5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400010e5a0, 0x68f90a, 0x40001cf8a8, 0x412bc0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400010e5a0, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40024c55f0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40024c55f0, 0x4000393000, 0x1000, 0x1000, 0x40001cf998, 0x44e884, 0x100) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40003d6d00, 0x4000393000, 0x1000, 0x1000, 0x4000180bf8, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40003d6d00, 0x4000393000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40003d6d40, 0x4000393000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40001cfd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001cfd80, 0xa, 0x4000393201, 0x1, 0xffff819ac998, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40001cfd80, 0x400039320a, 0x1, 0x4000119890, 0x0, 0x4000393259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40001cfd80, 0x40001b9b0a, 0x4000119890, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400012d080, 0x400038cc20, 0x1f, 0x40001cfee0, 0x400038e750, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x400012d080, 0x113075a, 0x7, 0x400041f0a0, 0xd, 0x400038e750, 0x0, 0x4714f4, 0x400012d080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x400012d040, 0x400041f0a0, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 73 [select]: net/http.(*persistConn).writeLoop(0x40001ae120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 101 [IO wait]: internal/poll.runtime_pollWait(0xffff8052a748, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001a4d18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40001a4d00, 0x400045f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40001a4d00, 0x400045f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011a268, 0x400045f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003d5200, 0x400045f000, 0x1000, 0x1000, 0x20, 0x30, 0x20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4001421500) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4001421500, 0x100a, 0x4000058bd8, 0x412ee0, 0x0, 0x0, 0x4000058c38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4001421500, 0x7356e4, 0xffffcfaef94e, 0xe, 0x58, 0x10786a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004567320) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004567320, 0x400472a600, 0x200, 0x200, 0xffffcfaef94e, 0xe, 0x4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400018f640, 0x400472a600, 0x200, 0x200, 0x4000038f07, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400018f640, 0x400472a600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400018f680, 0x400472a600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004727340, 0x471400, 0x4000058eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004727340, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004727340, 0xee0d40, 0x40001c48c0, 0x12db068, 0x400018f680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004571170, 0x400010fbc0, 0x4004576de0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 100 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x4000208460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 98 [select]: net/http.(*persistConn).readLoop(0x4000126ea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 58 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x40000827e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 59 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x4001424a50) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 60 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x4000154180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 61 [IO wait]: internal/poll.runtime_pollWait(0xffff8052a2c0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000154398, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000154380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000154380, 0x680da0, 0x400043a698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40024785b8, 0x40000389c0, 0x40003a9900, 0x40003e2f88) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40024785b8, 0x12ed9d0, 0x40024785b8, 0x58ec48, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x4000154280) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 62 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400142f450, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 63 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400142f450, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 64 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x400142f450) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 65 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x40014259b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 77 [select]: net/http.(*persistConn).writeLoop(0x40003d5200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 72 [IO wait]: internal/poll.runtime_pollWait(0xffff8052a1d8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000154598, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000154580, 0x4000459000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000154580, 0x4000459000, 0x1000, 0x1000, 0x40024b1be0, 0x469b10, 0x4000103200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011a228, 0x4000459000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001ae120, 0x4000459000, 0x1000, 0x1000, 0x40024b1cf8, 0x67867c, 0x400016b320) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40014211a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40014211a0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40001ae120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 145 [semacquire]: sync.runtime_Semacquire(0x4004e572c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004e572c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x4004653600, 0x40001825d0, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffcfaef969, 0x29, 0xffffcfaef9cb, 0x25, 0x4004c9ba40, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004cd2360, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffcfaef969, 0x29, 0xffffcfaef9cb, 0x25, 0x4004c9ba40, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004cd2360, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x400145f190, 0x400145ee60, 0x40001825d0, 0x40003e8d80, 0x400018e7c0, 0x0, 0xff, 0x4004dc8da8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x4001426000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004cd2360, 0xb, 0x4004cd2358, 0x6, 0x4004cd2370, 0xd, 0x4004c7b7c0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x4001426000, 0x40003a0000, 0x4001424030) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x4001426000, 0x400021d870, 0x2, 0x2, 0x400021d880, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 86 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000114c40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 119 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4004608c80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 120 [IO wait]: internal/poll.runtime_pollWait(0xffff8052a490, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000155818, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000155800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000155800, 0x58ec48, 0x400016b200, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c79da0, 0x4714f4, 0x400016b200, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c79da0, 0x400047a400, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x4004609580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 121 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x4004878000, 0x400141afc0, 0x400141b030, 0x4002473860, 0x12e8f90, 0x4002480a80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x40047fd674, 0x5, 0x4004609580, 0x12eca70, 0x4004609440, 0x130d478, 0x400027a770, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x4002480d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x4002480d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 87 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x40046a4780) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 88 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x40046a4780) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 89 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x4004878000, 0x4002473860) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 90 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x4002480a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 91 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x4002480a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 92 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x4002441050, 0x4004838b40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 93 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x4000114c00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 94 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x40046a4820) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 95 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x40046a4820) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 96 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x40000108f8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 97 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x40046a4820, 0x4004838ba0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 130 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400021c310, 0x1144559, 0x1b, 0x40003afe38, 0x4004838cc0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004da0380, 0x4004838cc0, 0xf1f700, 0x400021db90) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40003aff80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004d9e240, 0x4004da0380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 131 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 132 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400008d400, 0xd00007, 0x4004d92268, 0x5, 0x4004da0390, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 133 [IO wait]: internal/poll.runtime_pollWait(0xffff8052a0f0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1a18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f1a00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f1a00, 0xffffaac7c5b8, 0x30, 0x4004da4510) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004865f68, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004865f68, 0x4004da4510, 0x11ae500, 0x400046acc0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x400046ac40, 0x12ed9d0, 0x4004865f68, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x400046ac40, 0x12ed9d0, 0x4004865f68, 0x4001426000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 134 [IO wait]: internal/poll.runtime_pollWait(0xffff8052a008, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1b18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f1b00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f1b00, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004865f80, 0xffffaac7c5b8, 0xffff819d8f98, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004865f80, 0x665e1c, 0x0, 0xfe5e60, 0x40024ebdc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004865f98, 0x4004da45a0, 0x11ae500, 0x400046ada0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x400046ad20, 0x12ec710, 0x4004865f98, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x400046ad20, 0x12ec710, 0x4004865f98, 0x4001426000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 146 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x400046ae00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 141 [IO wait]: internal/poll.runtime_pollWait(0xffff80046940, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1e98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f1e80, 0x4004871000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f1e80, 0x4004871000, 0x1000, 0x1000, 0x40047c7be0, 0x469b10, 0x4004d95800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010990, 0x4004871000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40046e30e0, 0x4004871000, 0x1000, 0x1000, 0x40047c7cf8, 0x67867c, 0x4004839440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004d9ed20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004d9ed20, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40046e30e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 142 [select]: net/http.(*persistConn).writeLoop(0x40046e30e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 147 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x400046ae00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 148 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x400046ae00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 149 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x400046ae00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 150 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4004878740) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 162 [IO wait]: internal/poll.runtime_pollWait(0xffff80046a28, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004dee118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004dee100, 0x4004dfb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004dee100, 0x4004dfb000, 0x1000, 0x1000, 0x4004e87be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010a88, 0x4004dfb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40046e3320, 0x4004dfb000, 0x1000, 0x1000, 0x4004e87cf8, 0x67867c, 0x4004df27e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004dfc060) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004dfc060, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40046e3320) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 153 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4002403a20, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 154 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4002403a20, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 155 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4002403a20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 156 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x400046ae00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 157 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400021c310, 0x1144559, 0x1b, 0x4002499e38, 0x4004839aa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004da0a10, 0x4004839aa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4002499f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004d9f200, 0x4004da0a10) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 163 [select]: net/http.(*persistConn).writeLoop(0x40046e3320) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 160 [select]: net/http.(*persistConn).readLoop(0x40046e3440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 161 [select]: net/http.(*persistConn).writeLoop(0x40046e3440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 126 [IO wait]: internal/poll.runtime_pollWait(0xffff80046858, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004dee218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004dee200, 0x4004de7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004dee200, 0x4004de7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010a80, 0x4004de7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40046e3440, 0x4004de7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004d9ff20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004d9ff20, 0xa, 0x40023e0123, 0x11320c7, 0x9, 0x1, 0x2f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004d9ff20, 0x41ab8c, 0xffff80169f00, 0x100100000000, 0xffff802f8fff, 0x400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004ca3440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004ca3440, 0x4004e1ca00, 0x200, 0x200, 0x40024edd08, 0x41cff4, 0xffff80169f00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004609d00, 0x4004e1ca00, 0x200, 0x200, 0xffff8003c6f0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004609d00, 0x4004e1ca00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004609d40, 0x4004e1ca00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004e1e9a0, 0x0, 0x40024edeb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004e1e9a0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004e1e9a0, 0xee0d40, 0x4002484b40, 0x12db068, 0x4004609d40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004c95440, 0x4004ca08a0, 0x4004c9af60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 127 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4004878780) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 128 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x40024038c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 129 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x40024038c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 178 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x40024038c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 179 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x40024038c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 180 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x40047dfa00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 181 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x40047dfa00, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 182 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x4001426000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 183 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x4001426000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 184 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x4001426000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 185 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:50:45.854+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:50:45.854+00:00 [Info] Setting ipv6=false 2021-08-16T13:50:45.859+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:50:45.861+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:50:45.861+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:50:45.861+00:00 [Info] Setting log level to Info 2021-08-16T13:50:45.861+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:50:45.863+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:50:45 audit: created new audit service 2021-08-16T13:50:45.872+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:50:45.872+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:50:45.872+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:50:45.873+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:50:45.873+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:50:45.883+00:00 [Info] security context initialized 2021-08-16T13:50:45.883+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:50:45.883+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:50:45.883+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:50:45.883+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:50:45.883+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:50:45.896+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:50:45.901+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:50:45.902+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:50:45.922+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:50:45.924+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:50:45.925+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:50:45.926+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:50:45.929+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:50:45.929+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:50:45.931+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:50:45.935+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:50:45.977+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:50:45.977+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 54 us 2021-08-16T13:50:45.978+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:50:45.980+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:50:45.980+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:50:45.984+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:50:45.984+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:50:45.984+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:50:45.984+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:50:45.984+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:50:45.984+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:45.984+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:45.986+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:50:45.986+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:50:45.986+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:50:45.986+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:45.987+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:50:45.987+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:50:45.987+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:50:45.987+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:50:45.987+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:50:45.987+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:50:45.987+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:50:45.992+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 4.780781ms 2021-08-16T13:50:45.992+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:50:45.993+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:50:45.993+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:45.993+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:50:45.993+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:50:45.993+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:50:45.993+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:50:46.004+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:50:46.004+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:50:46.004+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:50:46.004+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:50:46.004+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x40004cbd5f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x40004cbd5f pc=0xbd8ea4] goroutine 188 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004e869f0 sp=0x4004e869c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004e86a30 sp=0x4004e869f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x4004479520, 0x4004c4e600, 0x11a9890, 0x4004e86b40, 0x4004479538, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004e86ab0 sp=0x4004e86a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x4004479520, 0x4004c4e600, 0x11a9890, 0x4004e86b40, 0x4004479538, 0xc18f8c, 0x4004e86c18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004e86af0 sp=0x4004e86ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x40044ec780, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004e86da0 sp=0x4004e86af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x40044ec780, 0x400446d490, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004e86e60 sp=0x4004e86da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004b5b970, 0x4004b5b980, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004e87200 sp=0x4004e86e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004b5b970, 0x4004b5b980, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004e878f0 sp=0x4004e87200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004c57670, 0x4004c39500, 0x40001cdb80, 0x4004b5b950) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004e87fb0 sp=0x4004e878f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004e87fb0 sp=0x4004e87fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x40000e2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x4000183560, 0x4, 0x1136a87, 0xf, 0x400011de68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 18 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400018e0f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 19 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400018e108) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 20 [IO wait]: internal/poll.runtime_pollWait(0xffff8c2b5758, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0180, 0x40000f9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0180, 0x40000f9000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000104f8, 0x40000f9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x4000082360, 0x40000fe001, 0xdff, 0xdff, 0x40000a18e8, 0x6902d8, 0x4000119948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400000e5b8, 0x40000fe001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40000ce840, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40000ce840, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40000ce840, 0xf5a960, 0x4000082440, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4000082420, 0x4000046340, 0x4000119b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400008c190, 0x12f8f98, 0x4000082420, 0x4000036ce7, 0x16, 0x400000d440, 0x40000a1c01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400008c190, 0x12f8f98, 0x4000082420, 0x400008c190, 0x40000364f0, 0x4000036500, 0x40000f0200, 0x4000046260, 0xee0980, 0x400000d440, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400008c190, 0x12f8f98, 0x4000082420) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x40001824e0, 0x40000a1f78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400009bf78, 0x40001824e0, 0x12db8a8, 0x400000e588, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x40001824e0, 0x40001a03c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x40001824e0, 0x40001a03c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 8 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400000a800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 21 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 22 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 23 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 24 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 25 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 26 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40004647a0, 0x4, 0x4000464668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40004647a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x40004647a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x40004647a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 32 [IO wait]: internal/poll.runtime_pollWait(0xffff8c2b5670, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001be118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40001be100, 0x40000ff000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40001be100, 0x40000ff000, 0x1000, 0x1000, 0x400006ba88, 0x68f7a4, 0x400019c890) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000186010, 0x40000ff000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c5a0, 0x40000ff000, 0x1000, 0x1000, 0xf12180, 0x400008c090, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003ecb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003ecb40, 0xa, 0x1008880, 0x400008c050, 0x199, 0xf90, 0x400006bb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003ecb40, 0x2, 0x0, 0x9, 0xa3c, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000430690) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000430690, 0x4000470000, 0xe00, 0xe00, 0x3d5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40000805c0, 0x4000470000, 0xe00, 0xe00, 0x3d5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40000805c0, 0x4000470000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40000815c0, 0x4000470000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x400019c840, 0x0, 0x400006beb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400019c840, 0x0, 0x0, 0x400006bf70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400019c840, 0xee0d40, 0x400008c050, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40000de090, 0x40003ecf60, 0x4000038360) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 30 [select]: net/http.(*persistConn).readLoop(0x400000c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 31 [select]: net/http.(*persistConn).writeLoop(0x400000c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 37 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000184730, 0x12c18bc, 0x1, 0x400009fe58, 0x40001b4900, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x40004676c8, 0x40001b4900, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400009ffa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x400018c000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 38 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x400018c000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 47 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400018d340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 41 [IO wait]: internal/poll.runtime_pollWait(0xffff8c2b5588, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001be218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40001be200, 0x4000172000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40001be200, 0x4000172000, 0x1000, 0x1000, 0x400048fbe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000120088, 0x4000172000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000ca20, 0x4000172000, 0x1000, 0x1000, 0x400048fcf8, 0x67867c, 0x40001b4f00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400010e900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x400010e900, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000ca20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 42 [select]: net/http.(*persistConn).writeLoop(0x400000ca20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 45 [select]: net/http.(*persistConn).readLoop(0x400000cfc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 46 [select]: net/http.(*persistConn).writeLoop(0x400000cfc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 113 [select]: net/http.(*persistConn).writeLoop(0x400249e5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 87 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400049d500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 14 [select]: net/http.(*persistConn).writeLoop(0x4002502240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 13 [select]: net/http.(*persistConn).readLoop(0x4002502240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 51 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x40023fe620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 52 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x40023fe620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 53 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x40023fe620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 54 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x40023fe620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 94 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x400440ca80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 66 [select]: net/http.(*persistConn).readLoop(0x40025027e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 67 [select]: net/http.(*persistConn).writeLoop(0x40025027e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 57 [IO wait]: internal/poll.runtime_pollWait(0xffff8c2b4f30, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0e98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0e80, 0x400049a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0e80, 0x400049a000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000498000, 0x400049a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40025027e0, 0x400049a000, 0x1000, 0x1000, 0x40001bdb00, 0x200000003, 0x40001bdb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000494060) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000494060, 0x400017600a, 0x40000a18c8, 0x4d3b10, 0x40000820c0, 0x4000176000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000494060, 0x0, 0x1, 0x0, 0x40000820d8, 0x4000176000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000496450) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000496450, 0x400446e000, 0x1000, 0x1000, 0x40000a19b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000080140, 0x400446e000, 0x1000, 0x1000, 0x400255cb98, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000080140, 0x400446e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000081640, 0x400446e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40000a1dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000a1dd0, 0xa, 0x400446e201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40000a1dd0, 0x400446e20a, 0x1, 0x40024bd4de, 0x0, 0x400446e2e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40000a1dd0, 0x40004a840a, 0x40024bd4de, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400250ad40, 0x4000190918, 0x17, 0x40000a1f20, 0x400000f530, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x400250ad40, 0x113075a, 0x7, 0x400000f530, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x400250ad00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 58 [IO wait]: internal/poll.runtime_pollWait(0xffff8c2b5018, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001bf218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40001bf200, 0x400030f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40001bf200, 0x400030f000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000120130, 0x400030f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000cfc0, 0x400030f000, 0x1000, 0x1000, 0x40001bdc80, 0x200000003, 0x40001bdc80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400010f500) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400010f500, 0x400017600a, 0x400009d8c8, 0x4d3b10, 0x40000820c0, 0x4000176000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400010f500, 0x0, 0x1, 0x0, 0x40000820d8, 0x4000176000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40003137a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40003137a0, 0x400011a000, 0x1000, 0x1000, 0x400009d9b8, 0x0, 0x400009d988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400018c6c0, 0x400011a000, 0x1000, 0x1000, 0x4000103380, 0x3, 0x400009dbe0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400018c6c0, 0x400011a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400018cbc0, 0x400011a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400009ddd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400009ddd0, 0xa, 0x400011a201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400009ddd0, 0x400011a20a, 0x1, 0x40024bd4db, 0x0, 0x400011a263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400009ddd0, 0x400012440a, 0x40024bd4db, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400250ad40, 0x4000112570, 0x24, 0x400009df20, 0x400000f518, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x400250ad40, 0x113075a, 0x7, 0x400000f518, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x400250ad00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 86 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400049c880) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 59 [IO wait]: internal/poll.runtime_pollWait(0xffff8c2b4e48, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001be298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40001be280, 0x4000328000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40001be280, 0x4000328000, 0x1000, 0x1000, 0x40003d3688, 0x69f178, 0x40003d36f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000120068, 0x4000328000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4002502240, 0x4000328000, 0x1000, 0x1000, 0x5, 0x40024e6000, 0x40003d37c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000480720) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000480720, 0x68f90a, 0x40004b46e0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000480720, 0x0, 0x40003d3868, 0x41a0a8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000110a50) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000110a50, 0x40024e2000, 0x1000, 0x1000, 0x40003d3998, 0x44e884, 0x1c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400049c480, 0x40024e2000, 0x1000, 0x1000, 0x400255cb98, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400049c480, 0x40024e2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400049c4c0, 0x40024e2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40003d3d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003d3d80, 0xa, 0x40024e2201, 0x1, 0xffffb5434db8, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40003d3d80, 0x40024e220a, 0x1, 0x400018b030, 0x0, 0x40024e2259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40003d3d80, 0x40004a3e0a, 0x400018b030, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400250ad40, 0x4004468160, 0x1f, 0x40003d3ee0, 0x400249a2d0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x400250ad40, 0x113075a, 0x7, 0x40004a6da0, 0xd, 0x400249a2d0, 0x0, 0x4714f4, 0x400250ad40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x400250ad00, 0x40004a6da0, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 112 [IO wait]: internal/poll.runtime_pollWait(0xffff8c2b4c78, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40024e0518, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40024e0500, 0x4000472000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40024e0500, 0x4000472000, 0x1000, 0x1000, 0x4002565be0, 0x469b10, 0x4000001e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010570, 0x4000472000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400249e5a0, 0x4000472000, 0x1000, 0x1000, 0x4002565cf8, 0x67867c, 0x40004a5860) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000083380) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000083380, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400249e5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 74 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40000e0070) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 72 [select]: net/http.(*persistConn).readLoop(0x400000d320) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 73 [select]: net/http.(*persistConn).writeLoop(0x400000d320) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 49 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4002464640) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 98 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x4000082ba0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 99 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x4000110ea0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 100 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x40024e0180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 101 [IO wait]: internal/poll.runtime_pollWait(0xffff8c2b4aa8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40024e0298, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40024e0280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40024e0280, 0x0, 0x4000339698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40004399b0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40004399b0, 0x12ed9d0, 0x40004399b0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x40024e0200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 102 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400032e820, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 103 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400032e820, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 104 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x400032e820) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 105 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x4000111f50) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 114 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4002464ac0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 75 [IO wait]: internal/poll.runtime_pollWait(0xffff8c2b4d60, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400447ac18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400447ac00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400447ac00, 0x12d9d48, 0x40000819c0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004b7cfd8, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004b7cfd8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x4004568b40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 76 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x400018de40, 0x400446cb60, 0x400446cbd0, 0x4000317170, 0x12e8f90, 0x40024f4a80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x4004685c34, 0x5, 0x4004568b40, 0x12eca70, 0x4002465640, 0x130d478, 0x40000e00e0, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x400440c8c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x400440c8c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 60 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x40000b1c20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 61 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x40000b1c20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 62 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x400018de40, 0x4000317170) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 63 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40024f4a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 64 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40024f4a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 65 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x40004b3170, 0x40000391a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 130 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x400049c840) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 131 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x40000b1cc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 132 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x40000b1cc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 133 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x4000186560) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 134 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x40000b1cc0, 0x4000039200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 135 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000184730, 0x1144559, 0x1b, 0x400014de38, 0x4000039320, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004bf8060, 0x4000039320, 0xf1f700, 0x4000185fb0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400014df80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40003edce0, 0x4004bf8060) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 136 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 137 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400008cfa0, 0xd00007, 0x4004af7748, 0x5, 0x4004bf8070, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 138 [IO wait]: internal/poll.runtime_pollWait(0xffff8c2b49c0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1298, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f1280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f1280, 0xffffb55b2a68, 0x30, 0x4004bfe0c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004bde348, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004bde348, 0x4004bfe0c0, 0x11ae500, 0x40023fe780, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40023fe700, 0x12ed9d0, 0x4004bde348, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40023fe700, 0x12ed9d0, 0x4004bde348, 0x40000e2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 139 [IO wait]: internal/poll.runtime_pollWait(0xffff8c2b48d8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1398, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f1380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f1380, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004bde360, 0xffffb55b2a68, 0xffff8c08e8f0, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004bde360, 0x665e1c, 0x0, 0xfe5e60, 0x40023ea5c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004bde378, 0x4004bfe150, 0x11ae500, 0x40023fe860, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40023fe7e0, 0x12ec710, 0x4004bde378, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40023fe7e0, 0x12ec710, 0x4004bde378, 0x40000e2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 95 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x400440ca80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 146 [IO wait]: internal/poll.runtime_pollWait(0xffff85582380, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1718, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f1700, 0x4004d78000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f1700, 0x4004d78000, 0x1000, 0x1000, 0x4000342be0, 0x469b10, 0x4004bd7c80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000186610, 0x4004d78000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40004c9e60, 0x4004d78000, 0x1000, 0x1000, 0x4000342cf8, 0x67867c, 0x4000039e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004bfcc00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004bfcc00, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40004c9e60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 147 [select]: net/http.(*persistConn).writeLoop(0x40004c9e60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 93 [semacquire]: sync.runtime_Semacquire(0x4004c57678) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004c57670) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x40001cdb80, 0x4000110c60, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffe4e64969, 0x29, 0xffffe4e649cb, 0x25, 0x4004c3c840, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004c012c0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffe4e64969, 0x29, 0xffffe4e649cb, 0x25, 0x4004c3c840, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004c012c0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004e87190, 0x4004e86e60, 0x4000110c60, 0x40001eb8c0, 0x400018d340, 0x0, 0x13f, 0x4002514da8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x40000e2000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004c012c0, 0xb, 0x4004c012b8, 0x6, 0x4004c012d0, 0xd, 0x4004b7f0a0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x40000e2000, 0x40003e0000, 0x4000110330) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x40000e2000, 0x40004907f0, 0x2, 0x2, 0x4000490800, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 78 [IO wait]: internal/poll.runtime_pollWait(0xffff8c2b4b90, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400447a218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400447a200, 0x40003f0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400447a200, 0x40003f0000, 0x1000, 0x1000, 0x1, 0x4004b5eed0, 0x4004c2da78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000498070, 0x40003f0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000d320, 0x40003f0000, 0x1000, 0x1000, 0x4004568d40, 0x0, 0x105f560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000494ba0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000494ba0, 0xa, 0x40004a0123, 0x11320c7, 0x9, 0x1, 0x13) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000494ba0, 0xd4fa28, 0x400003e200, 0x46, 0x11320c7, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004b5f020) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004b5f020, 0x4004c22200, 0x200, 0x200, 0x4004c013a0, 0x4004c013a0, 0x40004aa750) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004568dc0, 0x4004c22200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004568dc0, 0x4004c22200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004568e00, 0x4004c22200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004c04f20, 0x0, 0x400033beb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004c04f20, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004c04f20, 0xee0d40, 0x40004b08c0, 0x12db068, 0x4004568e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004c1e5a0, 0x4004c0a2a0, 0x4004567aa0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 96 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x400440ca80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 97 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x400440ca80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 162 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4002465c00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 176 [IO wait]: internal/poll.runtime_pollWait(0xffff85582298, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001bfc18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40001bfc00, 0x4004d8f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40001bfc00, 0x4004d8f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001204b0, 0x4004d8f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4002503560, 0x4004d8f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40046ff440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40046ff440, 0xa, 0x0, 0x0, 0x0, 0x0, 0x4000461c38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40046ff440, 0x0, 0x4000461c78, 0x655784, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004dc0570) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004dc0570, 0x40046e9600, 0x200, 0x200, 0x4, 0x0, 0x4000461d08) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4002465d80, 0x40046e9600, 0x200, 0x200, 0x4004db2217, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4002465d80, 0x40046e9600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4002465dc0, 0x40046e9600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004db8b00, 0x0, 0x4000461eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004db8b00, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004db8b00, 0xee0d40, 0x4000195040, 0x12db068, 0x4002465dc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004dbc5a0, 0x40046ff6e0, 0x4004db7080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 174 [IO wait]: internal/poll.runtime_pollWait(0xffff85582468, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001bfb18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40001bfb00, 0x4004dd1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40001bfb00, 0x4004dd1000, 0x1000, 0x1000, 0x4004bedbe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001204b8, 0x4004dd1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4002503440, 0x4004dd1000, 0x1000, 0x1000, 0x4004bedcf8, 0x67867c, 0x4004d7e420) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40046ff560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40046ff560, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4002503440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 165 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004db4420, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 166 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004db4420, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 167 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4004db4420) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 168 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x400440ca80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 169 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000184730, 0x1144559, 0x1b, 0x4004dcfe38, 0x4004db6360, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004dbe0e0, 0x4004db6360, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004dcff80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40046fe780, 0x4004dbe0e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 175 [select]: net/http.(*persistConn).writeLoop(0x4002503440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 172 [select]: net/http.(*persistConn).readLoop(0x4002503560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 173 [select]: net/http.(*persistConn).writeLoop(0x4002503560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 81 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4002465c40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 178 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x4004db42c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 179 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x4004db42c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 180 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x4004db42c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 181 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x4004db42c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 182 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004c06d00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 183 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x4004c06d00, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 184 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x40000e2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 185 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x40000e2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 186 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x40000e2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 187 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:50:51.157+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:50:51.157+00:00 [Info] Setting ipv6=false 2021-08-16T13:50:51.171+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:50:51.179+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:50:51.179+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:50:51.179+00:00 [Info] Setting log level to Info 2021-08-16T13:50:51.179+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:50:51.181+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:50:51 audit: created new audit service 2021-08-16T13:50:51.189+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:50:51.189+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:50:51.189+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:50:51.189+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:50:51.189+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:50:51.199+00:00 [Info] security context initialized 2021-08-16T13:50:51.199+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:50:51.199+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:50:51.199+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:50:51.199+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:50:51.199+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:50:51.213+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:50:51.213+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:50:51.226+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:50:51.238+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:50:51.238+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:50:51.240+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:50:51.241+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:50:51.243+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:50:51.246+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:50:51.246+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:50:51.246+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:50:51.290+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:50:51.291+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 53 us 2021-08-16T13:50:51.291+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:50:51.293+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:50:51.294+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:50:51.294+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:50:51.294+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:50:51.295+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:50:51.295+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:50:51.295+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:50:51.297+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:51.297+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:51.298+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:50:51.298+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:50:51.298+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:50:51.299+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:51.299+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:50:51.299+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:50:51.299+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:50:51.299+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:50:51.299+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:50:51.299+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:50:51.299+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:50:51.306+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 7.348934ms 2021-08-16T13:50:51.307+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:50:51.308+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:50:51.308+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:51.308+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:50:51.308+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:50:51.308+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:50:51.308+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:50:51.318+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:50:51.318+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:50:51.318+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:50:51.318+00:00 [Info] RebalanceMgr::initService Init 2021-08-16T13:50:51.318+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:50:51.318+00:00 [Info] ServiceMgr::registerWithServer nodeuuid 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:51.318+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x40003a569f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x40003a569f pc=0xbd8ea4] goroutine 186 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004e729f0 sp=0x4004e729c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004e72a30 sp=0x4004e729f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x400001b040, 0x4004e22600, 0x11a9890, 0x4004e72b40, 0x400001b058, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004e72ab0 sp=0x4004e72a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x400001b040, 0x4004e22600, 0x11a9890, 0x4004e72b40, 0x400001b058, 0x412ddc, 0xffff803f9618, 0x7f) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004e72af0 sp=0x4004e72ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x40001aedc0, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004e72da0 sp=0x4004e72af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x40001aedc0, 0x4000457180, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004e72e60 sp=0x4004e72da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004674c20, 0x4004674c30, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004e73200 sp=0x4004e72e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004674c20, 0x4004674c30, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004e738f0 sp=0x4004e73200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004e25840, 0x4004e58380, 0x40001ea100, 0x4004674c00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004e73fb0 sp=0x4004e738f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004e73fb0 sp=0x4004e73fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x4000434700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x4000216ed0, 0x4, 0x1136a87, 0xf, 0x4000095e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 20 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x40001aa630) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 21 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x40001aa648) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 22 [IO wait]: internal/poll.runtime_pollWait(0xffff8044be88, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000098118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000098100, 0x40000a7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000098100, 0x40000a7000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010030, 0x40000a7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x40000821e0, 0x40000a6001, 0xdff, 0xdff, 0x400019d8e8, 0x6902d8, 0x4000091948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400000e090, 0x40000a6001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40001e6000, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40001e6000, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40001e6000, 0xf5a960, 0x40003bae00, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x40003bade0, 0x40001c2240, 0x4000091b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x4000180050, 0x12f8f98, 0x40003bade0, 0x40001b5127, 0x16, 0x40003ea6c0, 0x400019dc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x4000180050, 0x12f8f98, 0x40003bade0, 0x4000180050, 0x40001b4190, 0x40001b41c0, 0x4000208000, 0x40001c2020, 0xee0980, 0x40003ea6c0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x4000180050, 0x12f8f98, 0x40003bade0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4000195e30, 0x400019df78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x40024adf78, 0x4000195e30, 0x12db8a8, 0x400000e060, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x4000195e30, 0x40001bd220, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x4000195e30, 0x40001bd220) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 23 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 24 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 25 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 26 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 27 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 34 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400010c200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 28 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40004057a0, 0x4, 0x4000405668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40004057a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x40001ba000, 0x40004057a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x40001b8000, 0x40004057a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 84 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000080800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 36 [select]: net/http.(*persistConn).readLoop(0x40003eafc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 37 [select]: net/http.(*persistConn).writeLoop(0x40003eafc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 31 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400022e2b0, 0x12c18bc, 0x1, 0x400019de58, 0x4000182c00, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x40004626c8, 0x4000182c00, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400019dfa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x40003ee680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 32 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x40003ee680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 15 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40023e8280) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 9 [IO wait]: internal/poll.runtime_pollWait(0xffff8044bcb8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000098198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000098180, 0x40000b8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000098180, 0x40000b8000, 0x1000, 0x1000, 0x4000067be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b86a0, 0x40000b8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c5a0, 0x40000b8000, 0x1000, 0x1000, 0x4000067cf8, 0x67867c, 0x4000182de0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003bb4a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40003bb4a0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 10 [select]: net/http.(*persistConn).writeLoop(0x400000c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 51 [IO wait]: internal/poll.runtime_pollWait(0xffff8044bda0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000208218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000208200, 0x4000134000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000208200, 0x4000134000, 0x1000, 0x1000, 0x4000138a88, 0x68f7a4, 0x4000472890) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010008, 0x4000134000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003eafc0, 0x4000134000, 0x1000, 0x1000, 0xf12180, 0x40001282c0, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000116540) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000116540, 0xa, 0x1008880, 0x4000128280, 0x199, 0x1000, 0x4000138b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000116540, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400008cbd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400008cbd0, 0x4000478000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40000802c0, 0x4000478000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40000802c0, 0x4000478000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000080300, 0x4000478000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4000472840, 0x0, 0x4000138eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4000472840, 0x0, 0x0, 0x4000138f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4000472840, 0xee0d40, 0x4000128280, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400009a990, 0x4000082840, 0x4000470420) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 92 [IO wait]: internal/poll.runtime_pollWait(0xffff803ab068, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400237ae18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400237ae00, 0x4004637000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400237ae00, 0x4004637000, 0x1000, 0x1000, 0x40000e0be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b87c8, 0x4004637000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40024730e0, 0x4004637000, 0x1000, 0x1000, 0x40000e0cf8, 0x67867c, 0x40046324e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40000d79e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40000d79e0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40024730e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 83 [select]: net/http.(*persistConn).writeLoop(0x4000162900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 42 [IO wait]: internal/poll.runtime_pollWait(0xffff8044b3a8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000098c98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000098c80, 0x4002429000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000098c80, 0x4002429000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b8660, 0x4002429000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000162900, 0x4002429000, 0x1000, 0x1000, 0x4000187e00, 0x200000003, 0x4000187e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40000d6c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000d6c00, 0x400014e00a, 0x400252b8f8, 0x412bc0, 0x40001ba060, 0x400014e000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40000d6c00, 0x0, 0x1, 0x0, 0x40001ba078, 0x400014e000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000216750) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000216750, 0x400257e000, 0x1000, 0x1000, 0x400252b9b8, 0x44d8a8, 0x80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40023e8400, 0x400257e000, 0x1000, 0x1000, 0x40001172b8, 0x400254e580, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40023e8400, 0x400257e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40023e8440, 0x400257e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400252bdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400252bdd0, 0xa, 0x400257e201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400252bdd0, 0x400257e20a, 0x1, 0x40001b57d9, 0x0, 0x400257e2e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400252bdd0, 0x40003cae0a, 0x40001b57d9, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000a8ba0, 0x400003a7f8, 0x17, 0x400252bf20, 0x4000475278, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x40000a8ba0, 0x113075a, 0x7, 0x4000475278, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x40000a8b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 54 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x400440c000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 93 [select]: net/http.(*persistConn).writeLoop(0x40024730e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 89 [IO wait]: internal/poll.runtime_pollWait(0xffff8044b2c0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400254e498, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400254e480, 0x40024a2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400254e480, 0x40024a2000, 0x1000, 0x1000, 0x400006ba48, 0x652ce0, 0x12edac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40003a21b0, 0x40024a2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400252e360, 0x40024a2000, 0x1000, 0x1000, 0x113674e, 0x412bc0, 0x40001282d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003baea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003baea0, 0x100a, 0x0, 0x0, 0x0, 0x0, 0x4000460c38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003baea0, 0x0, 0x4000460c78, 0x655784, 0x4000460ca8, 0x40b2f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004630000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004630000, 0x400247ea00, 0x200, 0x200, 0x4, 0x0, 0x4000460d08) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40003ef340, 0x400247ea00, 0x200, 0x200, 0x40001d5117, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40003ef340, 0x400247ea00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40003ef380, 0x400247ea00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004624f20, 0x680d00, 0x4000460eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004624f20, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004624f20, 0xee0d40, 0x4000180a50, 0x12db068, 0x40003ef380) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400462a5a0, 0x40000d7620, 0x4004632000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 178 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x4004c0edc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:662 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 60 [select]: net/http.(*persistConn).readLoop(0x400000ca20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 57 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x400440c000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 58 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x400440c000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 59 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x400440c000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 61 [select]: net/http.(*persistConn).writeLoop(0x400000ca20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 68 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000080380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 69 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x40023e4180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 70 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x40023782a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 71 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x400254e000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 72 [IO wait]: internal/poll.runtime_pollWait(0xffff8044b660, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400254e118, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400254e100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400254e100, 0x0, 0x4000058698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4002532288, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4002532288, 0x12ed9d0, 0x4002532288, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x400254e080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 73 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40003ca8f0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 74 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40003ca8f0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 75 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x40003ca8f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 76 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x4002379740) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 96 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x4004c0edc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:224 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 115 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x40004772c0, 0x4004695440) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 79 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40001f81c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 114 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000081580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 43 [IO wait]: internal/poll.runtime_pollWait(0xffff8044b578, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000098498, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000098480, 0x400047f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000098480, 0x400047f000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000120100, 0x400047f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000ca20, 0x400047f000, 0x1000, 0x1000, 0x400252c300, 0x200000003, 0x400252c300) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40000830e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000830e0, 0x400014e00a, 0x40025258c8, 0x4d3b10, 0x40001ba060, 0x400014e000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40000830e0, 0x0, 0x1, 0x0, 0x40001ba078, 0x400014e000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000195080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000195080, 0x400257b000, 0x1000, 0x1000, 0x40025259b8, 0x0, 0x4002525988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40023e8340, 0x400257b000, 0x1000, 0x1000, 0x4000001b00, 0x3, 0x4002525be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40023e8340, 0x400257b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40023e8380, 0x400257b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4002525dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002525dd0, 0xa, 0x400257b201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4002525dd0, 0x400257b20a, 0x1, 0x40001b558b, 0x0, 0x400257b263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4002525dd0, 0x400009e20a, 0x40001b558b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000a8ba0, 0x4000042cf0, 0x24, 0x4002525f20, 0x4000475260, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x40000a8ba0, 0x113075a, 0x7, 0x4000475260, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x40000a8b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 97 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x4004c0edc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:612 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 82 [select]: net/http.(*persistConn).readLoop(0x4000162900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 109 [select]: net/http.(*persistConn).readLoop(0x400252e360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 100 [IO wait]: internal/poll.runtime_pollWait(0xffff8044b1d8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400254e698, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400254e680, 0x40024a0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400254e680, 0x40024a0000, 0x1000, 0x1000, 0x40024a7688, 0x69f178, 0x40024a76f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40003a21a0, 0x40024a0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400252e5a0, 0x40024a0000, 0x1000, 0x1000, 0x5, 0x40024ba500, 0x40024a77c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40023e5f80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40023e5f80, 0x68f90a, 0x4000146840, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40023e5f80, 0x0, 0x2, 0x0, 0x40024a7878, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000118d50) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000118d50, 0x40024c7000, 0x1000, 0x1000, 0x40024a7998, 0x44e884, 0xffff00001000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40001265c0, 0x40024c7000, 0x1000, 0x1000, 0x4000083618, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40001265c0, 0x40024c7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000126600, 0x40024c7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40024a7d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40024a7d80, 0xa, 0x40024c7201, 0x1, 0xffff8045d350, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40024a7d80, 0x40024c720a, 0x1, 0x4002370b30, 0x0, 0x40024c7259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40024a7d80, 0x400235e00a, 0x4002370b30, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000a8ba0, 0x4002536500, 0x1f, 0x40024a7ee0, 0x4002532a20, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x40000a8ba0, 0x113075a, 0x7, 0x40001b57f0, 0xd, 0x4002532a20, 0x0, 0x0, 0x2) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x40000a8b60, 0x40001b57f0, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 110 [select]: net/http.(*persistConn).writeLoop(0x400252e360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 107 [select]: net/http.(*persistConn).readLoop(0x400252e5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 108 [select]: net/http.(*persistConn).writeLoop(0x400252e5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 142 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x4000200ee0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 141 [semacquire]: sync.runtime_Semacquire(0x4004e25848) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004e25840) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x40001ea100, 0x400008c5a0, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xfffff8fe4969, 0x29, 0xfffff8fe49cb, 0x25, 0x40046333e0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004c1cb20, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xfffff8fe4969, 0x29, 0xfffff8fe49cb, 0x25, 0x40046333e0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004c1cb20, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004e73190, 0x4004e72e60, 0x400008c5a0, 0x40003a4fc0, 0x40023e8280, 0x0, 0x17f, 0x40024aada8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x4000434700, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004c1cb20, 0xb, 0x4004c1cb18, 0x6, 0x4004c1cb30, 0xd, 0x4004c0bca0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x4000434700, 0x40003b0000, 0x4000217ec0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x4000434700, 0x40003ce180, 0x2, 0x2, 0x40003ce190, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 46 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40003ef180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 47 [IO wait]: internal/poll.runtime_pollWait(0xffff8044b490, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000099818, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000099800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000099800, 0x58ec48, 0x4000471b60, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c0c8e8, 0x4714f4, 0x4000471b60, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c0c8e8, 0x400011fe00, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x40001d8580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 48 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x4004c6c080, 0x4000211180, 0x40002111f0, 0x4004735b40, 0x12e8f90, 0x40002007e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x40046d3494, 0x5, 0x40001d8580, 0x12eca70, 0x4000127600, 0x130d478, 0x400027e9a0, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40000d07e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40000d07e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 49 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40002007e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 130 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40002007e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 131 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x400009b0e0, 0x400473d680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 132 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x40000807c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 133 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x40004772c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 134 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x40004772c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 135 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x4000120530) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 136 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x4000477360) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 137 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x4000477360) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 138 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x4004c6c080, 0x4004735b40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 116 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400022e2b0, 0x1144559, 0x1b, 0x4002501e38, 0x4004695560, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x40046d6cd0, 0x4004695560, 0x2c305b2c5d312d2c, 0x2d2c305b2c5d312d) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x2d2c305b2c5d312d, 0x312d2c305b2c5d31) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4002501f80, 0x312d2c305b2c5d31, 0x5d312d2c305b2c5d) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40046d49c0, 0x40046d6cd0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 117 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 118 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400015c910, 0xd00007, 0x40046d3820, 0x5, 0x40046d6ce0, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 119 [runnable]: internal/poll.runtime_pollWait(0xffff8044b748, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000209a18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000209a00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000209a00, 0xffffaa2ecf18, 0x30, 0x4004780a50) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40046f77e8, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40046f77e8, 0x4004780a50, 0x11ae500, 0x40000d0940, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40000d08c0, 0x12ed9d0, 0x40046f77e8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40000d08c0, 0x12ed9d0, 0x40046f77e8, 0x4000434700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 120 [IO wait]: internal/poll.runtime_pollWait(0xffff8044b008, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000209b18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000209b00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000209b00, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40046f7800, 0xffffaa2ecf18, 0xffff803fc1a8, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40046f7800, 0x665e1c, 0x0, 0xfe5e60, 0x40023c55c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x40046f7818, 0x4004780ae0, 0x11ae500, 0x40000d0a20, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40000d09a0, 0x12ec710, 0x40046f7818, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40000d09a0, 0x12ec710, 0x40046f7818, 0x4000434700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 122 [IO wait]: internal/poll.runtime_pollWait(0xffff803aaf80, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000209d98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000209d80, 0x4004cb0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000209d80, 0x4004cb0000, 0x1000, 0x1000, 0x40000dfbe0, 0x469b10, 0x4000103c80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010200, 0x4004cb0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400252f440, 0x4004cb0000, 0x1000, 0x1000, 0x40000dfcf8, 0x67867c, 0x40046958c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40046d4c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40046d4c00, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400252f440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 123 [select]: net/http.(*persistConn).writeLoop(0x400252f440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 95 [select]: net/http.(*persistConn).roundTrip(0x40024730e0, 0x4004c6ca80, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004dde700, 0xf8, 0x10a8660, 0x4004d62e18) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004dde700, 0x1b6e900, 0xc03eb94cd2fcb90a, 0x1bfa727a4b) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004dde600, 0x12db048, 0x1b6e900, 0xc03eb94cd2fcb90a, 0x1bfa727a4b, 0x1be4a80, 0x0, 0x40001206c0, 0x4004dc6b40, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004dc6b10, 0x4004dde600, 0xc03eb94cd2fcb90a, 0x1bfa727a4b, 0x1be4a80, 0x40001206c0, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004dc6b10, 0x4004dde600, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x40047418c0, 0x1b, 0x4004d63580, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004db6fc0, 0x112f905, 0x6, 0x12db908, 0x4000047ca0, 0xee40c0, 0x4004e8e018, 0x4004d63580, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x40001b2480, 0x46, 0x12db908, 0x4000047ca0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x40001b2480, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x20, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004d63d40, 0x4, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x4004c2eb40, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x4004c2eb40, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4004c6c640) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1267 +0xf8 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 94 [IO wait]: internal/poll.runtime_pollWait(0xffff8044b0f0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000099e98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000099e80, 0x4004dd0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000099e80, 0x4004dd0000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000120660, 0x4004dd0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003eb7a0, 0x4004dd0000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004dce600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004dce600, 0x100a, 0x0, 0x0, 0x0, 0x0, 0x4000461438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004dce600, 0x7356e4, 0x4000461478, 0x655784, 0x40004614a8, 0x40b2f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004630c30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004630c30, 0x400247f800, 0x200, 0x200, 0x4, 0x0, 0x4000461508) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40003ef880, 0x400247f800, 0x200, 0x200, 0x4004670007, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40003ef880, 0x400247f800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40003ef8c0, 0x400247f800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x400467eb00, 0x680d00, 0x40004616b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400467eb00, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400467eb00, 0xee0d40, 0x4000180cd0, 0x12db068, 0x40003ef8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400462af30, 0x40000d7e00, 0x4004632de0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 143 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x4000200ee0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 144 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x4000200ee0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 145 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x4000200ee0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 162 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4004c6c600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 163 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004c0ef20, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 164 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004c0ef20, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 165 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4004c0ef20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 166 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x4000200ee0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 167 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400022e2b0, 0x1144559, 0x1b, 0x4000199e38, 0x4004dc2180, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004db81d0, 0x4004dc2180, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000199f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40044e7aa0, 0x4004db81d0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 172 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).processSchedIndexes(0x4004c0edc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:296 +0x44 created by github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:218 +0x3c goroutine 170 [select]: net/http.(*persistConn).readLoop(0x40003eb7a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 171 [select]: net/http.(*persistConn).writeLoop(0x40003eb7a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 179 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x4004c0edc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:752 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 180 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004676900) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:249 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 177 [runnable]: internal/poll.runtime_pollWait(0xffff803aadb0, 0x77, 0x4004e5a300) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004e8e298, 0x77, 0x12f8600, 0x4004dcf500, 0x4004e8e280) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitWrite(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:96 internal/poll.(*FD).WaitWrite(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:528 net.(*netFD).connect(0x4004e8e280, 0x12f8668, 0x4004dcf500, 0x0, 0x0, 0x12db428, 0x4004e80520, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:141 +0x1e4 net.(*netFD).dial(0x4004e8e280, 0x12f8668, 0x4004dcf500, 0x12ff2c0, 0x0, 0x12ff2c0, 0x4004dc6f60, 0x0, 0xffff80484330, 0x10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:149 +0xac net.socket(0x12f8668, 0x4004dcf500, 0x112eab2, 0x3, 0x2, 0x1, 0x0, 0x0, 0x12ff2c0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:70 +0x148 net.internetSocket(0x12f8668, 0x4004dcf500, 0x112eab2, 0x3, 0x12ff2c0, 0x0, 0x12ff2c0, 0x4004dc6f60, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/ipsock_posix.go:141 +0xcc net.(*sysDialer).doDialTCP(0x4004e8e200, 0x12f8668, 0x4004dcf500, 0x0, 0x4004dc6f60, 0xf74f40, 0x1c1c218, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:65 +0x8c net.(*sysDialer).dialTCP(0x4004e8e200, 0x12f8668, 0x4004dcf500, 0x0, 0x4004dc6f60, 0x46e908, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:61 +0xb8 net.(*sysDialer).dialSingle(0x4004e8e200, 0x12f8668, 0x4004dcf500, 0x12e9300, 0x4004dc6f60, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:580 +0x4d4 net.(*sysDialer).dialSerial(0x4004e8e200, 0x12f8668, 0x4004dcf500, 0x4004db8b50, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:548 +0xf8 net.(*Dialer).DialContext(0x40001ba240, 0x12f8668, 0x4004dcf500, 0x112eab2, 0x3, 0x4004e88570, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:425 +0x550 net/http.(*Transport).dial(0x1b6e900, 0x12f8668, 0x4004dcf200, 0x112eab2, 0x3, 0x4004e88570, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1162 +0x178 net/http.(*Transport).dialConn(0x1b6e900, 0x12f8668, 0x4004dcf200, 0x0, 0x4004741b00, 0x4, 0x4004e88570, 0xe, 0x0, 0x40001c6480, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1600 +0x155c net/http.(*Transport).dialConnFor(0x1b6e900, 0x4004c0f600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1442 +0x90 created by net/http.(*Transport).queueForDial /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1411 +0x3a8 goroutine 182 [select]: net/http.(*Transport).getConn(0x1b6e900, 0x4004c6cac0, 0x0, 0x4004741a10, 0x4, 0x4004e88510, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1368 +0x418 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004dde900, 0xf8, 0x10a8660, 0x4004e94dd8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:579 +0x5c4 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004dde900, 0x1b6e900, 0xc03eb94cd2fdd92e, 0x1bfa739a6f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004dde800, 0x12db048, 0x1b6e900, 0xc03eb94cd2fdd92e, 0x1bfa739a6f, 0x1be4a80, 0x0, 0x40001206d0, 0x4004dc6cf0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004dc6cc0, 0x4004dde800, 0xc03eb94cd2fdd92e, 0x1bfa739a6f, 0x1be4a80, 0x40001206d0, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004dc6cc0, 0x4004dde800, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x40047419e0, 0x1b, 0x4004e95540, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004db7290, 0x112f905, 0x6, 0x12db908, 0x4000047d00, 0xee40c0, 0x4004e8e098, 0x4004e95540, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x40001b2580, 0x46, 0x12db908, 0x4000047d00, 0x4004dbb598, 0x81ee08, 0x4004dbb748, 0x95dcbc, 0x40001b2480, 0x46, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x40001b2580, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x240, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004e95d00, 0x4004dc6c00, 0xffff803a3b70) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x4004c2f200, 0xd47600, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x4004c2f200, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x4000434700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9702 +0x344 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 183 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x4000434700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1090 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 184 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x4000434700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1017 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 173 [runnable]: internal/poll.runtime_pollWait(0xffff803aae98, 0x77, 0x4004dc5200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004e8e198, 0x77, 0x12f8600, 0x4004dcf0e0, 0x4004e8e180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitWrite(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:96 internal/poll.(*FD).WaitWrite(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:528 net.(*netFD).connect(0x4004e8e180, 0x12f8668, 0x4004dcf0e0, 0x0, 0x0, 0x12db428, 0x4004e804c0, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:141 +0x1e4 net.(*netFD).dial(0x4004e8e180, 0x12f8668, 0x4004dcf0e0, 0x12ff2c0, 0x0, 0x12ff2c0, 0x4004dc6de0, 0x0, 0xffff80484330, 0x10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:149 +0xac net.socket(0x12f8668, 0x4004dcf0e0, 0x112eab2, 0x3, 0x2, 0x1, 0x0, 0x0, 0x12ff2c0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:70 +0x148 net.internetSocket(0x12f8668, 0x4004dcf0e0, 0x112eab2, 0x3, 0x12ff2c0, 0x0, 0x12ff2c0, 0x4004dc6de0, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/ipsock_posix.go:141 +0xcc net.(*sysDialer).doDialTCP(0x4004e8e100, 0x12f8668, 0x4004dcf0e0, 0x0, 0x4004dc6de0, 0xf74f40, 0x1c1c218, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:65 +0x8c net.(*sysDialer).dialTCP(0x4004e8e100, 0x12f8668, 0x4004dcf0e0, 0x0, 0x4004dc6de0, 0x46e908, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:61 +0xb8 net.(*sysDialer).dialSingle(0x4004e8e100, 0x12f8668, 0x4004dcf0e0, 0x12e9300, 0x4004dc6de0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:580 +0x4d4 net.(*sysDialer).dialSerial(0x4004e8e100, 0x12f8668, 0x4004dcf0e0, 0x4004db8a40, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:548 +0xf8 net.(*Dialer).DialContext(0x40001ba240, 0x12f8668, 0x4004dcf0e0, 0x112eab2, 0x3, 0x4004e88510, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:425 +0x550 net/http.(*Transport).dial(0x1b6e900, 0x12f8668, 0x4004dcede0, 0x112eab2, 0x3, 0x4004e88510, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1162 +0x178 net/http.(*Transport).dialConn(0x1b6e900, 0x12f8668, 0x4004dcede0, 0x0, 0x4004741a10, 0x4, 0x4004e88510, 0xe, 0x0, 0x40003ebe60, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1600 +0x155c net/http.(*Transport).dialConnFor(0x1b6e900, 0x4004c0f550) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1442 +0x90 created by net/http.(*Transport).queueForDial /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1411 +0x3a8 goroutine 185 [sleep]: time.Sleep(0x12a05f200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/plasma.RunMemQuotaTunerIteration(0x400045ffb0, 0xdf74d8, 0x4000182a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:211 +0x24c github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:247 +0xc0 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 goroutine 174 [select]: net.(*netFD).connect.func2(0x12f8668, 0x4004dcf0e0, 0x4004e8e180, 0x4004dc38c0, 0x4004dc3860) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:118 +0x70 created by net.(*netFD).connect /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:117 +0x1c0 goroutine 175 [runnable]: syscall.Syscall6(0xd1, 0x1d, 0x1, 0x4, 0x4004ea9480, 0x4004ea9478, 0x0, 0x0, 0x1, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:35 +0x10 syscall.getsockopt(0x1d, 0x1, 0x4, 0x4004ea9480, 0x4004ea9478, 0x570394, 0xffff803aacc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:1410 +0x64 syscall.GetsockoptInt(0x1d, 0x1, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:277 +0x54 net.(*netFD).connect(0x4004e8e380, 0x12f8630, 0x40001b4058, 0x0, 0x0, 0x12db428, 0x4004e80560, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:149 +0x224 net.(*netFD).dial(0x4004e8e380, 0x12f8630, 0x40001b4058, 0x12ff2c0, 0x0, 0x12ff2c0, 0x4004dc7050, 0x0, 0xffff80484330, 0x10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:149 +0xac net.socket(0x12f8630, 0x40001b4058, 0x112eab2, 0x3, 0x2, 0x1, 0x0, 0x0, 0x12ff2c0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:70 +0x148 net.internetSocket(0x12f8630, 0x40001b4058, 0x112eab2, 0x3, 0x12ff2c0, 0x0, 0x12ff2c0, 0x4004dc7050, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/ipsock_posix.go:141 +0xcc net.(*sysDialer).doDialTCP(0x4004e8e300, 0x12f8630, 0x40001b4058, 0x0, 0x4004dc7050, 0x0, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:65 +0x8c net.(*sysDialer).dialTCP(0x4004e8e300, 0x12f8630, 0x40001b4058, 0x0, 0x4004dc7050, 0xf89d40, 0x4004dc3bec, 0x1be4a80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:61 +0xb8 net.(*sysDialer).dialSingle(0x4004e8e300, 0x12f8630, 0x40001b4058, 0x12e9300, 0x4004dc7050, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:580 +0x4d4 net.(*sysDialer).dialSerial(0x4004e8e300, 0x12f8630, 0x40001b4058, 0x4004db8b90, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:548 +0xf8 net.(*Dialer).DialContext(0x4004ea9cb8, 0x12f8630, 0x40001b4058, 0x112eab2, 0x3, 0x4004dc3bec, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:425 +0x550 net.(*Dialer).Dial(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:348 net.Dial(0x112eab2, 0x3, 0x4004dc3bec, 0xe, 0x30, 0x1008940, 0x1, 0x4004dc6ff0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:319 +0x7c github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4004dc6ff0, 0x4004ea9f10, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:116 +0xc0 github.com/couchbase/cbauth/revrpc.BabysitService(0x4004ea9f10, 0x4004dc6ff0, 0x0, 0x0, 0x0, 0xd8d03c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth/service.RegisterManager(0x13007d8, 0x4004676900, 0x0, 0x0, 0x4004dbbf80, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:146 +0xa0 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).registerWithServer(0x4004676900) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:301 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:205 +0x78 goroutine 176 [select]: net/http.(*Transport).getConn(0x1b6e900, 0x4004c6cb00, 0x0, 0x4004741b00, 0x4, 0x4004e88570, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1368 +0x418 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004ddeb00, 0xf8, 0x10a8660, 0x4004dbb798) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:579 +0x5c4 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004ddeb00, 0x1b6e900, 0xc03eba5ad2ff7e3c, 0x1176f796f7d) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004ddea00, 0x12db048, 0x1b6e900, 0xc03eba5ad2ff7e3c, 0x1176f796f7d, 0x1be4a80, 0x0, 0x40001206e8, 0x4004dc6e70, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004dc6e40, 0x4004ddea00, 0xc03eba5ad2ff7e3c, 0x1176f796f7d, 0x1be4a80, 0x40001206e8, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004dc6e40, 0x4004ddea00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004e80500, 0x1f, 0x4004dbbd10, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/indexer.getWithAuth(0x4004e80500, 0x1f, 0xe, 0x113846e, 0x11) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:3077 +0x68 github.com/couchbase/indexing/secondary/indexer.getGlobalTopology(0x4004e24e60, 0xe, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:2266 +0x78 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).updateNodeList(0x4004676900) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:220 +0x3c created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:206 +0x98 goroutine 194 [select]: net.(*netFD).connect.func2(0x12f8668, 0x4004dcf500, 0x4004e8e280, 0x4004dc3b60, 0x4004dc3b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:118 +0x70 created by net.(*netFD).connect /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:117 +0x1c0 Initializing write barrier = 8000 2021-08-16T13:50:56.529+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:50:56.530+00:00 [Info] Setting ipv6=false 2021-08-16T13:50:56.538+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:50:56.544+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:50:56.544+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:50:56.544+00:00 [Info] Setting log level to Info 2021-08-16T13:50:56.544+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:50:56.547+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:50:56 audit: created new audit service 2021-08-16T13:50:56.554+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:50:56.555+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:50:56.555+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:50:56.555+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:50:56.555+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:50:56.566+00:00 [Info] security context initialized 2021-08-16T13:50:56.566+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:50:56.566+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:50:56.566+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:50:56.566+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:50:56.566+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:50:56.578+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:50:56.586+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:50:56.586+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:50:56.607+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:50:56.609+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:50:56.609+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:50:56.610+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:50:56.616+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:50:56.616+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:50:56.617+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:50:56.622+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:50:56.664+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:50:56.664+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 51 us 2021-08-16T13:50:56.665+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:50:56.667+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:50:56.667+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:50:56.668+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:50:56.668+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:50:56.668+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:50:56.668+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:50:56.668+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:50:56.674+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:56.675+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:56.677+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:50:56.677+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:50:56.677+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:50:56.677+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:56.678+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:50:56.678+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:50:56.678+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:50:56.679+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:50:56.679+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:50:56.679+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:50:56.679+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:50:56.682+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 3.327245ms 2021-08-16T13:50:56.683+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:50:56.684+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:50:56.684+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:56.684+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:50:56.684+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:50:56.684+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:50:56.684+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:50:56.694+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:50:56.694+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:50:56.694+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] unexpected fault address 0x4004bcafdf fatal error: fault 2021-08-16T13:50:56.695+00:00 [Info] RebalanceMgr::initService Init 2021-08-16T13:50:56.695+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:50:56.695+00:00 [Info] ServiceMgr::registerWithServer nodeuuid 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:50:56.695+00:00 [Info] : Recover shard shard1 from metadata completed. [signal SIGBUS: bus error code=0x1 addr=0x4004bcafdf pc=0xbd8ea4] goroutine 180 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x40024689f0 sp=0x40024689c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4002468a30 sp=0x40024689f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x4004bda4e0, 0x4004e6e000, 0x11a9890, 0x4002468b40, 0x4004bda4f8, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4002468ab0 sp=0x4002468a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x4004bda4e0, 0x4004e6e000, 0x11a9890, 0x4002468b40, 0x4004bda4f8, 0xc18f8c, 0x4002468c18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4002468af0 sp=0x4002468ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4004620640, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4002468da0 sp=0x4002468af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4004620640, 0x4001471960, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4002468e60 sp=0x4002468da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004e4e370, 0x4004e4e380, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4002469200 sp=0x4002468e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004e4e370, 0x4004e4e380, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x40024698f0 sp=0x4002469200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004e5c1f0, 0x4004e48e00, 0x400461d600, 0x4004e4e350) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4002469fb0 sp=0x40024698f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4002469fb0 sp=0x4002469fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x40001c2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x40001837d0, 0x4, 0x1136a87, 0xf, 0x4000119e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 18 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x40001aa0d8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 19 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x40001aa0f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 20 [IO wait]: internal/poll.runtime_pollWait(0xffffa2e2b1c8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001c0718, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40001c0700, 0x40001f5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40001c0700, 0x40001f5000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400018a0a0, 0x40001f5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x40001807e0, 0x40000f6001, 0xdff, 0xdff, 0x400009b8e8, 0x6902d8, 0x4000115948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x40001aa138, 0x40000f6001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40000ce840, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40000ce840, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40000ce840, 0xf5a960, 0x4000082380, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4000082360, 0x4000046300, 0x4000115b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400008c190, 0x12f8f98, 0x4000082360, 0x4000036c87, 0x16, 0x400000d320, 0x400009bc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400008c190, 0x12f8f98, 0x4000082360, 0x400008c190, 0x4000036498, 0x40000364a0, 0x40000f0080, 0x4000046220, 0xee0980, 0x400000d320, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400008c190, 0x12f8f98, 0x4000082360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x40001825d0, 0x400009bf78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400009ff78, 0x40001825d0, 0x12db8a8, 0x40001aa108, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x40001825d0, 0x40001a0460, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x40001825d0, 0x40001a0460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 21 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 8 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400000a700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 22 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 23 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 24 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 25 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 26 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40000557a0, 0x4, 0x4000055668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40000557a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x40000557a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x40000557a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 70 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x400001c2a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 32 [select]: net/http.(*persistConn).readLoop(0x40001ec120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 33 [select]: net/http.(*persistConn).writeLoop(0x40001ec120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 34 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001867f0, 0x12c18bc, 0x1, 0x400009de58, 0x40001b6900, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x40003f26c8, 0x40001b6900, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400009dfa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x400018c000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 35 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x400018c000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 40 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400042e480) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 38 [IO wait]: internal/poll.runtime_pollWait(0xffffa2e2aff8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000132118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000132100, 0x4001458000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000132100, 0x4001458000, 0x1000, 0x1000, 0x4001456be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011e078, 0x4001458000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c7e0, 0x4001458000, 0x1000, 0x1000, 0x4001456cf8, 0x67867c, 0x40000385a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400010e720) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x400010e720, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000c7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 39 [select]: net/http.(*persistConn).writeLoop(0x400000c7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 50 [IO wait]: internal/poll.runtime_pollWait(0xffffa2e2b0e0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001c0118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40001c0100, 0x40001f4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40001c0100, 0x40001f4000, 0x1000, 0x1000, 0x4001452a88, 0x68f7a4, 0x4000150050) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400018a048, 0x40001f4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001ec120, 0x40001f4000, 0x1000, 0x1000, 0xf12180, 0x4000152040, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400037d080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400037d080, 0xa, 0x1008880, 0x4000152000, 0x199, 0x1000, 0x4001452b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400037d080, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000400120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000400120, 0x400145a000, 0xe00, 0xe00, 0x4001452da8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000080000, 0x400145a000, 0xe00, 0xe00, 0x4001452e08, 0x40bb94, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000080000, 0x400145a000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000080140, 0x400145a000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4000150000, 0x0, 0x4001452eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4000150000, 0x0, 0x0, 0x4001452f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4000150000, 0xee0d40, 0x4000152000, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400013c000, 0x4000082420, 0x4000146000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 86 [select]: net/http.(*persistConn).readLoop(0x40001ed7a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 109 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x400045ad20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 51 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400018c540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 14 [IO wait]: internal/poll.runtime_pollWait(0xffffa2e2a7d0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001c0498, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40001c0480, 0x40000d1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40001c0480, 0x40000d1000, 0x1000, 0x1000, 0x40024f7688, 0x69f178, 0x40024f76f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400018a0b0, 0x40000d1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001ed8c0, 0x40000d1000, 0x1000, 0x1000, 0x5, 0x400449ca00, 0x40024f77c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400037de60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400037de60, 0x68f90a, 0x400013e210, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400037de60, 0x412ff8, 0x400441c840, 0x60, 0x58, 0x10786a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40024e4c90) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40024e4c90, 0x40000f3000, 0x1000, 0x1000, 0x40024f7998, 0x44e884, 0x10786a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400018cc40, 0x40000f3000, 0x1000, 0x1000, 0x40014724d8, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400018cc40, 0x40000f3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400018ccc0, 0x40000f3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40024f7d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40024f7d80, 0xa, 0x40000f3201, 0x1, 0xffffa2e8a828, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40024f7d80, 0x40000f320a, 0x1, 0x400241a420, 0x0, 0x40000f3259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40024f7d80, 0x4004406d0a, 0x400241a420, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000098ee0, 0x400247dd00, 0x1f, 0x40024f7ee0, 0x400000f548, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x4000098ee0, 0x113075a, 0x7, 0x40024d9310, 0xd, 0x400000f548, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x4000098ea0, 0x40024d9310, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 87 [select]: net/http.(*persistConn).writeLoop(0x40001ed7a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 12 [select]: net/http.(*persistConn).readLoop(0x400000d9e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 13 [select]: net/http.(*persistConn).writeLoop(0x400000d9e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 98 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4000420b60, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 76 [select]: net/http.(*persistConn).readLoop(0x4000163200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 73 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x400001c2a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 74 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x400001c2a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 75 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x400001c2a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 77 [select]: net/http.(*persistConn).writeLoop(0x4000163200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 53 [IO wait]: internal/poll.runtime_pollWait(0xffffa2e2aa88, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000132f18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000132f00, 0x400441e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000132f00, 0x400441e000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010860, 0x400441e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000163200, 0x400441e000, 0x1000, 0x1000, 0x400040c600, 0x200000003, 0x400040c600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400441c0c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400441c0c0, 0x400042400a, 0x40000a18c8, 0x4d3b10, 0x40000820c0, 0x4000424000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400441c0c0, 0x0, 0x1, 0x0, 0x40000820d8, 0x4000424000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40024e46c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40024e46c0, 0x400247e000, 0x1000, 0x1000, 0x40000a19b8, 0x0, 0x40000a1988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400018c400, 0x400247e000, 0x1000, 0x1000, 0x400040c300, 0x3, 0x40000a1be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400018c400, 0x400247e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400018c480, 0x400247e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40000a1dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000a1dd0, 0xa, 0x400247e201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40000a1dd0, 0x400247e20a, 0x1, 0x40024d92f9, 0x0, 0x400247e2e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40000a1dd0, 0x400019550a, 0x40024d92f9, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000098ee0, 0x400041a768, 0x17, 0x40000a1f20, 0x40004141e0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x4000098ee0, 0x113075a, 0x7, 0x40004141e0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x4000098ea0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 54 [IO wait]: internal/poll.runtime_pollWait(0xffffa2e2a9a0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0398, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0380, 0x4002454000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0380, 0x4002454000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011e1d0, 0x4002454000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000d9e0, 0x4002454000, 0x1000, 0x1000, 0x400040c780, 0x200000003, 0x400040c780) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400010f380) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400010f380, 0x400042400a, 0x40024f9858, 0x4344e0, 0x40000820c0, 0x4000424000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400010f380, 0x41c330, 0xffffa2e9dc40, 0x0, 0x0, 0x4000424000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40024e4ae0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40024e4ae0, 0x40000d0000, 0x1000, 0x1000, 0x40024f99b8, 0x44d8a8, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400018c600, 0x40000d0000, 0x1000, 0x1000, 0x400010e358, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400018c600, 0x40000d0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400018c640, 0x40000d0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40024f9dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40024f9dd0, 0xa, 0x40000d0201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40024f9dd0, 0x40000d020a, 0x1, 0x40024d961b, 0x0, 0x40000d0263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40024f9dd0, 0x400018e20a, 0x40024d961b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000098ee0, 0x4000418120, 0x24, 0x40024f9f20, 0x40004141c8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x4000098ee0, 0x113075a, 0x7, 0x40004141c8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x4000098ea0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 84 [select]: net/http.(*persistConn).readLoop(0x40001ed8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 85 [select]: net/http.(*persistConn).writeLoop(0x40001ed8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 80 [runnable]: internal/poll.runtime_pollWait(0xffffa2e2a6e8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000132198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000132180, 0x400443e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000132180, 0x400443e000, 0x1000, 0x1000, 0x4004420be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010378, 0x400443e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400043e000, 0x400443e000, 0x1000, 0x1000, 0x4004420cf8, 0x67867c, 0x4000422ba0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400441c660) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x400441c660, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400043e000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 81 [select]: net/http.(*persistConn).writeLoop(0x400043e000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 41 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x4000082b40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 42 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x400246c6c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 43 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x40000f0280) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 44 [IO wait]: internal/poll.runtime_pollWait(0xffffa2e2a600, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0418, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f0400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f0400, 0x312d2c305b2c5d31, 0x4000148e98, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004514510, 0x5d312d2c305b2c5d, 0x2c5d312d2c305b00, 0x5b2c5d312d2c305b) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004514510, 0x12ed9d0, 0x4004514510, 0x305b2c5d312d2c30, 0x2c305b2c5d312d2c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x40000f0300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 99 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4000420b60, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 100 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x4000420b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 101 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x4004404bd0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 57 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400012d540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 110 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x400045ad20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 104 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40001fe8c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 111 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x400042f780, 0x40024cea30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 45 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400018d100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 47 [runnable]: internal/poll.runtime_pollWait(0xffff78557430, 0x77, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1118, 0x77, 0x4000036000, 0x1, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitWrite(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:96 internal/poll.(*FD).WaitWrite(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:528 net.(*netFD).connect(0x40000f1100, 0x12f8630, 0x4000036098, 0x0, 0x0, 0x12db428, 0x40046cb200, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:141 +0x1e4 net.(*netFD).dial(0x40000f1100, 0x12f8630, 0x4000036098, 0x12ff2c0, 0x0, 0x12ff2c0, 0x400469a5d0, 0x0, 0xffff785fea08, 0x10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:149 +0xac net.socket(0x12f8630, 0x4000036098, 0x112eab2, 0x3, 0x2, 0x1, 0x0, 0x0, 0x12ff2c0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:70 +0x148 net.internetSocket(0x12f8630, 0x4000036098, 0x112eab2, 0x3, 0x12ff2c0, 0x0, 0x12ff2c0, 0x400469a5d0, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/ipsock_posix.go:141 +0xcc net.(*sysDialer).doDialTCP(0x40000f1080, 0x12f8630, 0x4000036098, 0x0, 0x400469a5d0, 0x0, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:65 +0x8c net.(*sysDialer).dialTCP(0x40000f1080, 0x12f8630, 0x4000036098, 0x0, 0x400469a5d0, 0xf89d40, 0x400468510c, 0x41a0a8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:61 +0xb8 net.(*sysDialer).dialSingle(0x40000f1080, 0x12f8630, 0x4000036098, 0x12e9300, 0x400469a5d0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:580 +0x4d4 net.(*sysDialer).dialSerial(0x40000f1080, 0x12f8630, 0x4000036098, 0x40046945e0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:548 +0xf8 net.(*Dialer).DialContext(0x4004bf7cb8, 0x12f8630, 0x4000036098, 0x112eab2, 0x3, 0x400468510c, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:425 +0x550 net.(*Dialer).Dial(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:348 net.Dial(0x112eab2, 0x3, 0x400468510c, 0xe, 0x30, 0x1008940, 0x1, 0x400469a570) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:319 +0x7c github.com/couchbase/cbauth/revrpc.(*Service).Run(0x400469a570, 0x4004bf7f10, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:116 +0xc0 github.com/couchbase/cbauth/revrpc.BabysitService(0x4004bf7f10, 0x400469a570, 0x0, 0x0, 0x0, 0xd8d03c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth/service.RegisterManager(0x13007d8, 0x400474f300, 0x0, 0x0, 0x4001454f80, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:146 +0xa0 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).registerWithServer(0x400474f300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:301 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:205 +0x78 goroutine 46 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400012cec0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 58 [IO wait]: internal/poll.runtime_pollWait(0xffffa2e2a518, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400450e798, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400450e780, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400450e780, 0x12d9d48, 0x400146e700, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004b81320, 0x0, 0x400041a768, 0x17) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004b81320, 0x40004141e0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x400146f780) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 59 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x400042f780, 0x4001470e70, 0x4001470ee0, 0x40024cea30, 0x12e8f90, 0x400013a700, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x400473fd54, 0x5, 0x400146f780, 0x12eca70, 0x400146f600, 0x130d478, 0x40000e0150, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40044b2700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40044b2700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 60 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x400013a700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 61 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x400013a700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 62 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x4000429680, 0x4004707440) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 63 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x400018d0c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 64 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x40001a1900) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 65 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x40001a1900) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 114 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x400043c4c8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 115 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x40001a1900, 0x40047074a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 116 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001867f0, 0x1144559, 0x1b, 0x4000155e38, 0x40047075c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004b837a0, 0x40047075c0, 0xf1f700, 0x40003ee070) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000155f80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4001473f20, 0x4004b837a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 117 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 118 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x4000196eb0, 0xd00007, 0x4004bab7b8, 0x5, 0x4004b837b0, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 119 [runnable]: internal/poll.runtime_pollWait(0xffffa2e2a430, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400450ea18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400450ea00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400450ea00, 0xffffa2f1d5b8, 0x30, 0x4004bbcd20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004b818c0, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004b818c0, 0x4004bbcd20, 0x11ae500, 0x40044b2860, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40044b27e0, 0x12ed9d0, 0x4004b818c0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40044b27e0, 0x12ed9d0, 0x4004b818c0, 0x40001c2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 120 [IO wait]: internal/poll.runtime_pollWait(0xffffa2e2a348, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400450eb18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400450eb00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400450eb00, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004b818d8, 0xffffa2f1d5b8, 0xffffa2ea95a0, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004b818d8, 0x665e1c, 0x0, 0xfe5e60, 0x400452d5c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004b818f0, 0x4004bbcdb0, 0x11ae500, 0x40044b2940, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40044b28c0, 0x12ec710, 0x4004b818f0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40044b28c0, 0x12ec710, 0x4004b818f0, 0x40001c2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 96 [IO wait]: internal/poll.runtime_pollWait(0xffffa2e2a8b8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001c0398, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40001c0380, 0x40000ef000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40001c0380, 0x40000ef000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400018a150, 0x40000ef000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001ed7a0, 0x40000ef000, 0x1000, 0x1000, 0x20, 0x30, 0x20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400037df20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400037df20, 0x100a, 0xffffa2e9f208, 0x8, 0x0, 0x55, 0x40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400037df20, 0x41ab8c, 0xffff7bb0e600, 0x100000000000, 0xffff7bceefff, 0x400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40003c8600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40003c8600, 0x40003c7200, 0x200, 0x200, 0x400014f508, 0x41cff4, 0xffff7bb0e600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400018d800, 0x40003c7200, 0x200, 0x200, 0xffffa2e89040, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400018d800, 0x40003c7200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400018d840, 0x40003c7200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40003ec000, 0x680d00, 0x400014f6b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40003ec000, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40003ec000, 0xee0d40, 0x400008cbe0, 0x12db068, 0x400018d840) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40000df0e0, 0x400010fe60, 0x40003ea000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 141 [select]: net/http.(*persistConn).writeLoop(0x4004bc06c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 124 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40044b2b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 134 [IO wait]: internal/poll.runtime_pollWait(0xffff78557600, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000133918, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000133900, 0x4004d5c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000133900, 0x4004d5c000, 0x1000, 0x1000, 0x4000452be0, 0x469b10, 0x400445a180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010920, 0x4004d5c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400043f8c0, 0x4004d5c000, 0x1000, 0x1000, 0x4000452cf8, 0x67867c, 0x4002491ce0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40024c5200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40024c5200, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400043f8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 135 [select]: net/http.(*persistConn).writeLoop(0x400043f8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 123 [semacquire]: sync.runtime_Semacquire(0x4004e5c1f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004e5c1f0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x400461d600, 0x4000110510, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffe021f969, 0x29, 0xffffe021f9cb, 0x25, 0x4004becba0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004baaaf0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffe021f969, 0x29, 0xffffe021f9cb, 0x25, 0x4004becba0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004baaaf0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4002469190, 0x4002468e60, 0x4000110510, 0x4002496000, 0x400018c540, 0x0, 0xffffa2f281f8, 0x78, 0xffff78552af8, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x40001c2000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004baaaf0, 0xb, 0x4004baaae8, 0x6, 0x4004baab00, 0xd, 0x400477f280, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x40001c2000, 0x4000370000, 0x40003ac5d0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x40001c2000, 0x400011c560, 0x2, 0x2, 0x400011c570, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 125 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40044b2b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 126 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40044b2b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 127 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40044b2b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 128 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x400146ff00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 129 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004bea2c0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 162 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004bea2c0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 163 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4004bea2c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 164 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40044b2b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 165 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001867f0, 0x1144559, 0x1b, 0x4004bd1e38, 0x4004707c20, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004b83bf0, 0x4004707c20, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004bd1f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004bde7e0, 0x4004b83bf0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 142 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).processSchedIndexes(0x4004bea160) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:296 +0x44 created by github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:218 +0x3c goroutine 140 [select]: net/http.(*persistConn).readLoop(0x4004bc06c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 168 [IO wait]: internal/poll.runtime_pollWait(0xffff785576e8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400450f098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400450f080, 0x4004d67000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400450f080, 0x4004d67000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010978, 0x4004d67000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004bc06c0, 0x4004d67000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40024c5980) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40024c5980, 0x55b60a, 0xf1f980, 0x0, 0x37, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40024c5980, 0x0, 0x1, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004bbd8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004bbd8c0, 0x4004b87400, 0x200, 0x200, 0x2, 0xe, 0x4001447628) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40000805c0, 0x4004b87400, 0x200, 0x200, 0xffffe021f9cb, 0x0, 0x4000110510) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40000805c0, 0x4004b87400, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40000816c0, 0x4004b87400, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004b8bce0, 0x0, 0x4000059eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004b8bce0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004b8bce0, 0xee0d40, 0x40001971d0, 0x12db068, 0x40000816c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004bcc990, 0x4004bdf140, 0x4004bec5a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 169 [runnable]: strings.(*Builder).grow(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/strings/builder.go:69 strings.(*Builder).Grow(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/strings/builder.go:82 net/url.unescape(0x4004b94338, 0xe, 0x3, 0x1, 0x9, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/url/url.go:253 +0x870 net/url.parseHost(0x4004b94338, 0xe, 0x12c1e70, 0x1, 0x30, 0x4004de0870) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/url/url.go:662 +0x27c net/url.parseAuthority(0x4004b94307, 0x3f, 0x2f, 0x4004b94307, 0x3f, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/url/url.go:587 +0x84 net/url.parse(0x4004b94300, 0x46, 0x100, 0x4004b94300, 0x46, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/url/url.go:567 +0x21c net/url.Parse(0x4004b94300, 0x46, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/url/url.go:478 +0x70 github.com/couchbase/indexing/secondary/dcp.ParseURL(0x4004b94300, 0x46, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/util.go:43 +0x30 github.com/couchbase/indexing/secondary/dcp.basicAuthFromURL(0x4004b94300, 0x46, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:512 +0x30 github.com/couchbase/indexing/secondary/dcp.Connect(0x4004b94300, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x54 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x20, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004bd3d40, 0x4, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x4004bca900, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x4004bca900, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400146ff40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1267 +0xf8 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 170 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x4004bea160) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:224 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 171 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x4004bea160) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:612 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 172 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x4004bea160) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:662 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 173 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x4004bea160) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:752 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 174 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x400474f300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:249 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 49 [runnable]: internal/poll.runtime_pollWait(0xffff78557518, 0x77, 0x4004e58600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1018, 0x77, 0x12f8600, 0x40045c8fc0, 0x40000f1000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitWrite(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:96 internal/poll.(*FD).WaitWrite(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:528 net.(*netFD).connect(0x40000f1000, 0x12f8668, 0x40045c8fc0, 0x0, 0x0, 0x12db428, 0x40046cb1c0, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:141 +0x1e4 net.(*netFD).dial(0x40000f1000, 0x12f8668, 0x40045c8fc0, 0x12ff2c0, 0x0, 0x12ff2c0, 0x400469a4e0, 0x0, 0xffff785fea08, 0x10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:149 +0xac net.socket(0x12f8668, 0x40045c8fc0, 0x112eab2, 0x3, 0x2, 0x1, 0x0, 0x0, 0x12ff2c0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/sock_posix.go:70 +0x148 net.internetSocket(0x12f8668, 0x40045c8fc0, 0x112eab2, 0x3, 0x12ff2c0, 0x0, 0x12ff2c0, 0x400469a4e0, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/ipsock_posix.go:141 +0xcc net.(*sysDialer).doDialTCP(0x40000f0f80, 0x12f8668, 0x40045c8fc0, 0x0, 0x400469a4e0, 0xf74f40, 0x1c1c218, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:65 +0x8c net.(*sysDialer).dialTCP(0x40000f0f80, 0x12f8668, 0x40045c8fc0, 0x0, 0x400469a4e0, 0x46e908, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:61 +0xb8 net.(*sysDialer).dialSingle(0x40000f0f80, 0x12f8668, 0x40045c8fc0, 0x12e9300, 0x400469a4e0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:580 +0x4d4 net.(*sysDialer).dialSerial(0x40000f0f80, 0x12f8668, 0x40045c8fc0, 0x40046945a0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:548 +0xf8 net.(*Dialer).DialContext(0x40000822a0, 0x12f8668, 0x40045c8fc0, 0x112eab2, 0x3, 0x40046b2680, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/dial.go:425 +0x550 net/http.(*Transport).dial(0x1b6e900, 0x12f8668, 0x40045c8de0, 0x112eab2, 0x3, 0x40046b2680, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1162 +0x178 net/http.(*Transport).dialConn(0x1b6e900, 0x12f8668, 0x40045c8de0, 0x0, 0x400012fef0, 0x4, 0x40046b2680, 0xe, 0x0, 0x400000db00, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1600 +0x155c net/http.(*Transport).dialConnFor(0x1b6e900, 0x40000c5ef0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1442 +0x90 created by net/http.(*Transport).queueForDial /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1411 +0x3a8 goroutine 176 [select]: net/http.(*persistConn).roundTrip(0x400043e000, 0x400012d840, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004696700, 0xf8, 0x10a8660, 0x40024f2dd8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004696700, 0x1b6e900, 0xc03eb94e296e0a2b, 0x1bfab0cf3a) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004696600, 0x12db048, 0x1b6e900, 0xc03eb94e296e0a2b, 0x1bfab0cf3a, 0x1be4a80, 0x0, 0x400011e280, 0x400469a300, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x400469a2d0, 0x4004696600, 0xc03eb94e296e0a2b, 0x1bfab0cf3a, 0x1be4a80, 0x400011e280, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x400469a2d0, 0x4004696600, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x400012fdd0, 0x1b, 0x40024f3540, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x40045d75f0, 0x112f905, 0x6, 0x12db908, 0x4000128cc0, 0xee40c0, 0x40000f0f18, 0x40024f3540, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x4002448200, 0x46, 0x12db908, 0x4000128cc0, 0xffff7bc13a00, 0x100000000000, 0xffff7bceef00, 0x400, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x4002448200, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x240, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40024f3d00, 0x400469a210, 0xffffa2f27fd8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x40000b98c0, 0xd47600, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x40000b98c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x40001c2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9702 +0x344 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 177 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x40001c2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1090 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 178 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x40001c2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1017 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 179 [runnable]: time.Since(0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/time/time.go:868 +0x9c github.com/couchbase/plasma.RunMemQuotaTunerIteration(0x4001454fb0, 0xdf74d8, 0x40001b6780) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:202 +0x6c4 github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:247 +0xc0 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 goroutine 48 [select]: net/http.(*Transport).getConn(0x1b6e900, 0x400012d880, 0x0, 0x400012fef0, 0x4, 0x40046b2680, 0xe, 0x0, 0x0, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1368 +0x418 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004696900, 0xf8, 0x10a8660, 0x4001454798) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:579 +0x5c4 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004696900, 0x1b6e900, 0xc03eba5c296f5c56, 0x1176fb65165) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004696800, 0x12db048, 0x1b6e900, 0xc03eba5c296f5c56, 0x1176fb65165, 0x1be4a80, 0x0, 0x400011e290, 0x400469a3f0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x400469a3c0, 0x4004696800, 0xc03eba5c296f5c56, 0x1176fb65165, 0x1be4a80, 0x400011e290, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x400469a3c0, 0x4004696800, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x40046cb1a0, 0x1f, 0x4001454d10, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/indexer.getWithAuth(0x40046cb1a0, 0x1f, 0xe, 0x113846e, 0x11) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:3077 +0x68 github.com/couchbase/indexing/secondary/indexer.getGlobalTopology(0x4004bff810, 0xe, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:2266 +0x78 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).updateNodeList(0x400474f300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:220 +0x3c created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:206 +0x98 goroutine 194 [select]: net.(*netFD).connect.func2(0x12f8668, 0x40045c8fc0, 0x40000f1000, 0x4004685080, 0x4004685020) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:118 +0x70 created by net.(*netFD).connect /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:117 +0x1c0 Initializing write barrier = 8000 2021-08-16T13:51:01.856+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:51:01.856+00:00 [Info] Setting ipv6=false 2021-08-16T13:51:01.868+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:51:01.870+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:51:01.870+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:51:01.870+00:00 [Info] Setting log level to Info 2021-08-16T13:51:01.870+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:51:01.873+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:51:01 audit: created new audit service 2021-08-16T13:51:01.880+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:51:01.881+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:51:01.881+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:51:01.881+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:51:01.881+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:51:01.897+00:00 [Info] security context initialized 2021-08-16T13:51:01.898+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:51:01.898+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:51:01.898+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:51:01.898+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:51:01.898+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:51:01.910+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:51:01.911+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:51:01.916+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:51:01.938+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:51:01.939+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:51:01.939+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:51:01.941+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:51:01.943+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:51:01.944+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:51:01.944+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:51:01.950+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:51:01.996+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:51:01.996+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 52 us 2021-08-16T13:51:01.997+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:51:01.999+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:51:02.000+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:51:02.000+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:51:02.000+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:51:02.000+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:51:02.000+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:51:02.000+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:51:02.003+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:02.003+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:02.004+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:51:02.004+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:51:02.004+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:51:02.004+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:02.006+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:51:02.006+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:51:02.007+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:51:02.007+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:51:02.008+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:51:02.008+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:51:02.009+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:51:02.012+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 4.041439ms 2021-08-16T13:51:02.012+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:51:02.014+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:51:02.014+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:02.014+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:51:02.014+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:51:02.014+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:51:02.014+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:51:02.024+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:51:02.024+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:51:02.024+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:51:02.024+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:51:02.024+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 2021-08-16T13:51:02.024+00:00 [Info] RebalanceMgr::initService Init 2021-08-16T13:51:02.024+00:00 [Info] ServiceMgr::registerWithServer nodeuuid 8a95cde683482bdec453edbb8b18eb5d 0x400246ed9f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x400246ed9f pc=0xbd8ea4] goroutine 184 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004d229f0 sp=0x4004d229c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004d22a30 sp=0x4004d229f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x40044cf380, 0x40046a9200, 0x11a9890, 0x4004d22b40, 0x40044cf398, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004d22ab0 sp=0x4004d22a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x40044cf380, 0x40046a9200, 0x11a9890, 0x4004d22b40, 0x40044cf398, 0xffffad3c4e50, 0x4004d22c18, 0x49) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004d22af0 sp=0x4004d22ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x40003daa00, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004d22da0 sp=0x4004d22af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x40003daa00, 0x40044b2fc0, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004d22e60 sp=0x4004d22da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004b76ce0, 0x4004b76cf0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004d23200 sp=0x4004d22e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004b76ce0, 0x4004b76cf0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004d238f0 sp=0x4004d23200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004e9aff0, 0x40046d1c00, 0x4000419600, 0x4004b76cc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004d23fb0 sp=0x4004d238f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004d23fb0 sp=0x4004d23fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x4000146000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x40000fcf60, 0x4, 0x1136a87, 0xf, 0x4000125e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 7 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400000e660) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 8 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400000e678) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 9 [IO wait]: internal/poll.runtime_pollWait(0xffff8406ce88, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000190118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000190100, 0x40001b7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000190100, 0x40001b7000, 0x1000, 0x1000, 0x1311860, 0x47b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400018a018, 0x40001b7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x4000186240, 0x40001dc001, 0xdff, 0xdff, 0x400009b8e8, 0x6902d8, 0x4000121948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x4000198030, 0x40001dc001, 0xdff, 0xdff, 0x40001219f8, 0x6f1968, 0x4000121938) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40001b4160, 0xa, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40001b4160, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40001b4160, 0xf5a960, 0x40001866e0, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x40001866c0, 0x400018c1c0, 0x4000121b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x40001c01e0, 0x12f8f98, 0x40001866c0, 0x400018e8c7, 0x16, 0x40001aa480, 0x400009bc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x40001c01e0, 0x12f8f98, 0x40001866c0, 0x40001c01e0, 0x400018e0f0, 0x400018e100, 0x4000190180, 0x400018c0e0, 0xee0980, 0x40001aa480, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x40001c01e0, 0x12f8f98, 0x40001866c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4000097ec0, 0x400009bf78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x4001449f78, 0x4000097ec0, 0x12db8a8, 0x4000198000, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x4000097ec0, 0x40000b15e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x4000097ec0, 0x40000b15e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 10 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 11 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 12 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 13 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 14 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 18 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x40001be100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 15 [syscall]: syscall.Syscall(0x3f, 0x0, 0x4000054fa0, 0x4, 0x4000054e68, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x4000054fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x4000054fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x4000054fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 106 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x40001c4aa0, 0x4004b9a720) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 53 [IO wait]: internal/poll.runtime_pollWait(0xffff8406cda0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400012e198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400012e180, 0x40001dd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400012e180, 0x40001dd000, 0x1000, 0x1000, 0x400006aa88, 0x68f7a4, 0x40001b4b50) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000128028, 0x40001dd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001aa120, 0x40001dd000, 0x1000, 0x1000, 0xf12180, 0x400008c3b0, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001163c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001163c0, 0xa, 0x1008880, 0x400008c370, 0x199, 0x1000, 0x400006ab00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40001163c0, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000118d20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000118d20, 0x400144e000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400012c240, 0x400144e000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400012c240, 0x400144e000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400012c280, 0x400144e000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40001b4b00, 0x0, 0x400006aeb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40001b4b00, 0x0, 0x0, 0x400006af70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40001b4b00, 0xee0d40, 0x400008c370, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40001a6d80, 0x4000116780, 0x4000144c60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 51 [select]: net/http.(*persistConn).readLoop(0x40001aa120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 52 [select]: net/http.(*persistConn).writeLoop(0x40001aa120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 66 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40002142e0, 0x12c18bc, 0x1, 0x4001415e58, 0x40001446c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x4000138ec8, 0x40001446c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4001415fa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x400012c040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 67 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x400012c040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 123 [IO wait]: internal/poll.runtime_pollWait(0xffff8406c490, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1718, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f1700, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f1700, 0x58ec48, 0x400456e540, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40023d9278, 0x4714f4, 0x400456e540, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40023d9278, 0x40023dcf00, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x4000185540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 70 [IO wait]: internal/poll.runtime_pollWait(0xffff8406ccb8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000190098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000190080, 0x400144c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000190080, 0x400144c000, 0x1000, 0x1000, 0x40001a1be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010378, 0x400144c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003d47e0, 0x400144c000, 0x1000, 0x1000, 0x40001a1cf8, 0x67867c, 0x4000038840) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003a1440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40003a1440, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40003d47e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 71 [select]: net/http.(*persistConn).writeLoop(0x40003d47e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 58 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000184700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 56 [select]: net/http.(*persistConn).readLoop(0x40001aafc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 57 [select]: net/http.(*persistConn).writeLoop(0x40001aafc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 76 [IO wait]: internal/poll.runtime_pollWait(0xffffad272618, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000191418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000191400, 0x4004c01000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000191400, 0x4004c01000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400018a648, 0x4004c01000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400247b440, 0x4004c01000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004cc2d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004cc2d80, 0x100a, 0x0, 0x0, 0x0, 0x0, 0x4000134c38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004cc2d80, 0xede200, 0x4000134c78, 0x655784, 0x4000134ca8, 0x40b2f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40024ad020) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40024ad020, 0x40046dea00, 0x200, 0x200, 0x60, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4002400dc0, 0x40046dea00, 0x200, 0x200, 0x40001453e0, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4002400dc0, 0x40046dea00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4002400e00, 0x40046dea00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004b78420, 0x680d00, 0x4000134eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004b78420, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004b78420, 0xee0d40, 0x40044b8640, 0x12db068, 0x4002400e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40046bb0e0, 0x40044ab9e0, 0x40024ab380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 60 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x4000448540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 86 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4000098f70, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 85 [IO wait]: internal/poll.runtime_pollWait(0xffff8406c748, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400012eb98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400012eb80, 0x400044e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400012eb80, 0x400044e000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000128128, 0x400044e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001aafc0, 0x400044e000, 0x1000, 0x1000, 0x4000103b00, 0x200000003, 0x4000103b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001172c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001172c0, 0x400145200a, 0x400144b8c8, 0x4d3b10, 0x40000820c0, 0x4001452000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40001172c0, 0x0, 0x1, 0x0, 0x40000820d8, 0x4001452000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000460810) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000460810, 0x4002502000, 0x1000, 0x1000, 0x400144b9b8, 0x0, 0x400144b988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400012c300, 0x4002502000, 0x1000, 0x1000, 0x4000103200, 0x3, 0x400144bbe0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400012c300, 0x4002502000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400012c340, 0x4002502000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400144bdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400144bdd0, 0xa, 0x4002502201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400144bdd0, 0x400250220a, 0x1, 0x400253d49b, 0x0, 0x4002502263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400144bdd0, 0x400014a10a, 0x400253d49b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40025385f0, 0x400011a120, 0x24, 0x400144bf20, 0x4002552a20, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x40025385f0, 0x113075a, 0x7, 0x4002552a20, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x40025385b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 63 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x4000448540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 64 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x4000448540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 65 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x4000448540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 84 [IO wait]: internal/poll.runtime_pollWait(0xffff8406c3a8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400012e418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400012e400, 0x400442d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400012e400, 0x400442d000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000128088, 0x400442d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000cc60, 0x400442d000, 0x1000, 0x1000, 0x40003dc180, 0x200000003, 0x40003dc180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002536ba0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002536ba0, 0x400145200a, 0x400256f8c8, 0x4d3b10, 0x40000820c0, 0x4001452000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4002536ba0, 0x0, 0x1, 0x0, 0x40000820d8, 0x4001452000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400449c480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400449c480, 0x4004554000, 0x1000, 0x1000, 0x400256f9b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000184380, 0x4004554000, 0x1000, 0x1000, 0x40001871f8, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000184380, 0x4004554000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001843c0, 0x4004554000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400256fdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400256fdd0, 0xa, 0x4004554201, 0x1, 0xd1f, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400256fdd0, 0x400455420a, 0x1, 0x40024cb039, 0x0, 0x40045542e1, 0x1, 0xd1f, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400256fdd0, 0x400009900a, 0x40024cb039, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40025385f0, 0x400003a450, 0x17, 0x400256ff20, 0x4002552a50, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x40025385f0, 0x113075a, 0x7, 0x4002552a50, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x40025385b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 40 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400012d100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 107 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40002142e0, 0x1144559, 0x1b, 0x4000171e38, 0x4004b9a840, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004679240, 0x4004b9a840, 0x400013a73c, 0x40001c3338) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x4002506000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000171f80, 0x12c2e00, 0x100010000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004700f00, 0x4004679240) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 22 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001846c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 41 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x4000186780) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 42 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x40001a4120) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 43 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x40000f0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 44 [IO wait]: internal/poll.runtime_pollWait(0xffff8406c578, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0198, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f0180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f0180, 0x0, 0x400013be98, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40024fdce0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40024fdce0, 0x12ed9d0, 0x40024fdce0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x40000f0100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 87 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4000098f70, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 88 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x4000098f70) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 89 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x40004602a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 109 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400008d180, 0xd00007, 0x40047750a8, 0x5, 0x4004679250, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 108 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 124 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x4000185980, 0x4000012af0, 0x4000012b60, 0x4004c96690, 0x12e8f90, 0x40045d8460, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x40023f6284, 0x5, 0x4000185540, 0x12eca70, 0x4000185400, 0x130d478, 0x40000e0150, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40000e8540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40000e8540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 95 [select]: net/http.(*persistConn).readLoop(0x400000cc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 94 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x4000272a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 96 [select]: net/http.(*persistConn).writeLoop(0x400000cc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 101 [select]: net/http.(*persistConn).readLoop(0x400000db00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 102 [select]: net/http.(*persistConn).writeLoop(0x400000db00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 47 [IO wait]: internal/poll.runtime_pollWait(0xffff8406c1d8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0498, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0480, 0x4004556000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0480, 0x4004556000, 0x1000, 0x1000, 0x400256b688, 0x69f178, 0x400256b6f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400018a150, 0x4004556000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003d45a0, 0x4004556000, 0x1000, 0x1000, 0x5, 0x4000472000, 0x400256b7c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000187560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000187560, 0x68f90a, 0x4000016630, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000187560, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000461c80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000461c80, 0x4002572000, 0x1000, 0x1000, 0x400256b998, 0x44e884, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400012c500, 0x4002572000, 0x1000, 0x1000, 0x4000116898, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400012c500, 0x4002572000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400012c540, 0x4002572000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400256bd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400256bd80, 0xa, 0x4002572201, 0x1, 0xffffad245210, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400256bd80, 0x400257220a, 0x1, 0x4002436b80, 0x0, 0x4002572259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400256bd80, 0x400450170a, 0x4002436b80, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40025385f0, 0x40024feb20, 0x1f, 0x400256bee0, 0x4000198120, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x40025385f0, 0x113075a, 0x7, 0x40024cb050, 0xd, 0x4000198120, 0x0, 0x0, 0x2) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x40025385b0, 0x40024cb050, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 105 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40003cf000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 114 [select]: net/http.(*persistConn).readLoop(0x40003d45a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 115 [select]: net/http.(*persistConn).writeLoop(0x40003d45a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 122 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400012da80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 196 [select]: net/http.(*persistConn).writeLoop(0x400247b560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 163 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40002142e0, 0x1144559, 0x1b, 0x4001419e38, 0x4004b9b800, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x400042b200, 0x4004b9b800, 0x4000055eb8, 0x6a1ed4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1008880, 0x4000462601) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4001419f80, 0x12db068, 0x40003cf880) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004701e00, 0x400042b200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 125 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40045d8460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 126 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40045d8460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 127 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x40024c0fc0, 0x400456f080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 128 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x4000184680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 129 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x40001c4aa0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 130 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x40001c4aa0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 131 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x400018a598) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 132 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x40001c4b40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 133 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x40001c4b40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 134 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x4000185980, 0x4004c96690) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 110 [IO wait]: internal/poll.runtime_pollWait(0xffff8406c0f0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400012fa98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400012fa80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400012fa80, 0x0, 0x4004bc2d88, 0x665fe4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004bb6360, 0x611a6d46, 0x4004bc2db8, 0x445c24) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004bb6360, 0x4004bc2e20, 0x4004bc2e28, 0x18, 0x40003ddb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40045d89a0, 0x12ed9d0, 0x4004bb6360, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40045d89a0, 0x12ed9d0, 0x4004bb6360, 0x4000146000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 111 [IO wait]: internal/poll.runtime_pollWait(0xffff8406c008, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400012fb98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400012fb80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400012fb80, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004bb6378, 0xffffad3bb108, 0xffff7c99e118, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004bb6378, 0x665e1c, 0x0, 0xfe5e60, 0x4000137dc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004bb6390, 0x4004bb8750, 0x11ae500, 0x40045d8b00, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40045d8a80, 0x12ec710, 0x4004bb6390, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40045d8a80, 0x12ec710, 0x4004bb6390, 0x4000146000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 162 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40045d8ee0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 30 [IO wait]: internal/poll.runtime_pollWait(0xffffad2727e8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000191218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000191200, 0x4004d2c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000191200, 0x4004d2c000, 0x1000, 0x1000, 0x40044a6be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000107f8, 0x4004d2c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400247b320, 0x4004d2c000, 0x1000, 0x1000, 0x40044a6cf8, 0x67867c, 0x4000423bc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001db320) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40001db320, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400247b320) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 26 [IO wait]: internal/poll.runtime_pollWait(0xffffad272700, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000191118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000191100, 0x4004c7e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000191100, 0x4004c7e000, 0x1000, 0x1000, 0x400249bbe0, 0x469b10, 0x4004cc0600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000107b8, 0x4004c7e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004cb2a20, 0x4004c7e000, 0x1000, 0x1000, 0x400249bcf8, 0x67867c, 0x40004237a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001daf00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40001daf00, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004cb2a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 27 [select]: net/http.(*persistConn).writeLoop(0x4004cb2a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 113 [IO wait]: internal/poll.runtime_pollWait(0xffff8406c2c0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400012e798, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400012e780, 0x4002570000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400012e780, 0x4002570000, 0x1000, 0x1000, 0x1, 0x4004bb8960, 0x4004bc4a78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001280f8, 0x4002570000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000db00, 0x4002570000, 0x1000, 0x1000, 0x4004b9c580, 0x0, 0x105f560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40025370e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40025370e0, 0xa, 0x4002430123, 0x11320c7, 0x9, 0x1, 0x4000137ce4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40025370e0, 0xd4fa28, 0x400003e380, 0x46, 0x11320c7, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004bb8d20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004bb8d20, 0x4004bea200, 0x200, 0x200, 0x40047751b0, 0x40047751b0, 0x4000042600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004b9c780, 0x4004bea200, 0x200, 0x200, 0x0, 0xffffad3bb108, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004b9c780, 0x4004bea200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004b9c7c0, 0x4004bea200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004bcac60, 0x12ec700, 0x4000137eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004bcac60, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004bcac60, 0xee0d40, 0x400008d3b0, 0x12db068, 0x4004b9c7c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40047790e0, 0x40047016e0, 0x4004b9af00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 152 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40045d8ee0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 151 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40045d8ee0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 150 [semacquire]: sync.runtime_Semacquire(0x4004e9aff8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004e9aff0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x4000419600, 0x40000fc450, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffca64a969, 0x29, 0xffffca64a9cb, 0x25, 0x40024abc20, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x40023f6f70, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffca64a969, 0x29, 0xffffca64a9cb, 0x25, 0x40024abc20, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x40023f6f70, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004d23190, 0x4004d22e60, 0x40000fc450, 0x40000b9440, 0x4000184700, 0x0, 0xbf, 0x4001416da8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x4000146000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x40023f6f70, 0xb, 0x40023f6f68, 0x6, 0x40023f6f80, 0xd, 0x40023d6d40, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x4000146000, 0x40003a0000, 0x40001181b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x4000146000, 0x40000a0190, 0x2, 0x2, 0x40000a01a0, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 31 [select]: net/http.(*persistConn).writeLoop(0x400247b320) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 142 [select]: net/http.(*persistConn).writeLoop(0x400247b440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 153 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40045d8ee0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 154 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40045d8ee0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 155 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4004b9cbc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 156 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004713080, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 157 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004713080, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 158 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4004713080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 141 [select]: net/http.(*persistConn).readLoop(0x400247b440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 77 [select]: net/http.(*persistConn).roundTrip(0x400247b320, 0x4004e4c080, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004ceed00, 0xf8, 0x10a8660, 0x400256ce18) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004ceed00, 0x1b6e900, 0xc03eb94f8178f053, 0x1bfae5c33f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004ceec00, 0x12db048, 0x1b6e900, 0xc03eb94f8178f053, 0x1bfae5c33f, 0x1be4a80, 0x0, 0x400018a6a8, 0x4004e52210, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004e521e0, 0x4004ceec00, 0xc03eb94f8178f053, 0x1bfae5c33f, 0x1be4a80, 0x400018a6a8, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004e521e0, 0x4004ceec00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004e06480, 0x1b, 0x400256d580, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004cbf200, 0x112f905, 0x6, 0x12db908, 0x400018d580, 0xee40c0, 0x40000f1b98, 0x400256d580, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x4004704380, 0x46, 0x12db908, 0x400018d580, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x4004704380, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x20, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400256dd40, 0x4, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x4002558fc0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x4002558fc0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4004b9cc00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1267 +0xf8 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 78 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x4004712f20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:224 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 79 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x4004712f20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:612 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 80 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x4004712f20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:662 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 81 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x4004712f20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:752 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 178 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004b7a600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:249 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 197 [select]: net/http.(*persistConn).roundTrip(0x4004cb30e0, 0x4004e4c340, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004cef200, 0xf8, 0x10a8660, 0x4004cf4588) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004cef200, 0x1b6e900, 0xc03eb94f817da061, 0x1bfaea7355) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004cef100, 0x12db048, 0x1b6e900, 0xc03eb94f817da061, 0x1bfaea7355, 0x1be4a80, 0x0, 0x400018a6e8, 0x4004e52930, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004e52900, 0x4004cef100, 0xc03eb94f817da061, 0x1bfaea7355, 0x1be4a80, 0x400018a6e8, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004e52900, 0x4004cef100, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004e066f0, 0x1b, 0x4004cf4cf0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004cbf950, 0x112f905, 0x6, 0x12db908, 0x400018d620, 0xee40c0, 0x40000f1e98, 0x4004cf4cf0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x40023e6580, 0x46, 0x12db908, 0x400018d620, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x40023e6580, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x4004e52840, 0xffff7cc99680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004e754b0, 0x1063520, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x4004cbcd80, 0x4004cbcd80, 0x40023e6580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.FetchNewClusterInfoCache(0xffffca64a94e, 0xe, 0x113075a, 0x7, 0x1130e44, 0x8, 0x4004e630c4, 0x2, 0xf01) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:123 +0x134 github.com/couchbase/indexing/secondary/manager.(*IndexManager).FetchNewClusterInfoCache(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:306 github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).getIndexMetadata(0x1be5520, 0x12ec7a0, 0x4004e4c300, 0x400001d270, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:1284 +0x74 github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).handleIndexMetadataRequest(0x1be5520, 0x12edbe0, 0x40000e8700, 0x4004cef000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:1271 +0x3e8 net/http.HandlerFunc.ServeHTTP(0x4004679140, 0x12edbe0, 0x40000e8700, 0x4004cef000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2069 +0x40 net/http.(*ServeMux).ServeHTTP(0x4004b9c380, 0x12edbe0, 0x40000e8700, 0x4004cef000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2448 +0x190 net/http.serverHandler.ServeHTTP(0x40045d89a0, 0x12edbe0, 0x40000e8700, 0x4004cef000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2887 +0xbc net/http.(*conn).serve(0x40001c5220, 0x12f86a0, 0x4004e4c180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:1952 +0x71c created by net/http.(*Server).Serve /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:3013 +0x318 goroutine 180 [runnable]: net/http.(*persistConn).roundTrip(0x4004cb2fc0, 0x4004e4c0c0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004ceef00, 0xf8, 0x10a8660, 0x400016edd8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004ceef00, 0x1b6e900, 0xc03eb94f817a114c, 0x1bfae6e440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004ceee00, 0x12db048, 0x1b6e900, 0xc03eb94f817a114c, 0x1bfae6e440, 0x1be4a80, 0x0, 0x400018a6b8, 0x4004e523c0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004e52390, 0x4004ceee00, 0xc03eb94f817a114c, 0x1bfae6e440, 0x1be4a80, 0x400018a6b8, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004e52390, 0x4004ceee00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004e065a0, 0x1b, 0x400016f540, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004cbf4d0, 0x112f905, 0x6, 0x12db908, 0x400018d5e0, 0xee40c0, 0x40000f1c18, 0x400016f540, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x40023e6500, 0x46, 0x12db908, 0x400018d5e0, 0x4004d37598, 0x81ee08, 0x4004d37748, 0x95dcbc, 0x4004704380, 0x46, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x40023e6500, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x240, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400016fd00, 0x4004e522d0, 0xffff7cc99680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x4004cbcb40, 0xd47600, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x4004cbcb40, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x4000146000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9702 +0x344 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 181 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x4000146000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1090 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 182 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x4000146000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1017 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 183 [sleep]: time.Sleep(0x12a05f200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/plasma.RunMemQuotaTunerIteration(0x40025617b0, 0xdf74d8, 0x4000144540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:211 +0x24c github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:247 +0xc0 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 goroutine 166 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).processSchedIndexes(0x4004712f20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:296 +0x44 created by github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:218 +0x3c goroutine 167 [IO wait]: internal/poll.runtime_pollWait(0xffffad272360, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000191618, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000191600, 0x4004d60000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000191600, 0x4004d60000, 0x1000, 0x1000, 0x0, 0x4004d35000, 0x4004cf7a38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010868, 0x4004d60000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).fill(0x4004d5e060) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004d5e060, 0x40a, 0x4004d60000, 0x0, 0xffff00001001, 0x400, 0x400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).ReadLine(0x4004d5e060, 0x1000, 0xffff7cc37d38, 0x1, 0x4004cf7bc8, 0x412ff8, 0x4004d50480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:389 +0x30 net/textproto.(*Reader).readLineSlice(0x4004cf7cf0, 0x4004d50480, 0x4004cf7c48, 0xffff7cc37d38, 0x2c8, 0xffffad3bba90) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/textproto/reader.go:57 +0xac net/textproto.(*Reader).ReadLine(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/textproto/reader.go:38 net/http.ReadResponse(0x4004d5e060, 0x4000459500, 0x1000, 0x4004d60000, 0x4000010868) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/response.go:161 +0x8c github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4000427ce0, 0x4004cf7f10, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:132 +0x2ac github.com/couchbase/cbauth/revrpc.BabysitService(0x4004cf7f10, 0x4000427ce0, 0x0, 0x0, 0x0, 0xd8d03c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth/service.RegisterManager(0x13007d8, 0x4004b7a600, 0x0, 0x0, 0x4004d3bf80, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:146 +0xa0 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).registerWithServer(0x4004b7a600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:301 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:205 +0x78 goroutine 168 [select]: net/http.(*persistConn).roundTrip(0x400247b560, 0x40003cf980, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4000459400, 0xf8, 0x10a8660, 0x4002494798) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4000459400, 0x1b6e900, 0xc03eba5d8179c872, 0x1176feacb66) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4000459300, 0x12db048, 0x1b6e900, 0xc03eba5d8179c872, 0x1176feacb66, 0x1be4a80, 0x0, 0x4000010858, 0x4000427b60, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4000427b30, 0x4000459300, 0xc03eba5d8179c872, 0x1176feacb66, 0x1be4a80, 0x4000010858, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4000427b30, 0x4000459300, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4000428900, 0x1f, 0x4002494d10, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/indexer.getWithAuth(0x4000428900, 0x1f, 0xe, 0x113846e, 0x11) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:3077 +0x68 github.com/couchbase/indexing/secondary/indexer.getGlobalTopology(0x4004e9a610, 0xe, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:2266 +0x78 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).updateNodeList(0x4004b7a600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:220 +0x3c created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:206 +0x98 goroutine 195 [IO wait]: internal/poll.runtime_pollWait(0xffff8406c660, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000191518, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000191500, 0x4004e61000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000191500, 0x4004e61000, 0x1000, 0x1000, 0x0, 0x0, 0x4002566c78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400018a6d0, 0x4004e61000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400247b560, 0x4004e61000, 0x1000, 0x1000, 0x60, 0x0, 0x12be201) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004cc3a40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004cc3a40, 0x1, 0x4004e1eae0, 0xe, 0x1c1c218, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400247b560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 145 [select]: net/http.(*persistConn).readLoop(0x4004cb2fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 194 [select]: net/http.(*persistConn).writeLoop(0x4004cb2fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 198 [IO wait]: internal/poll.runtime_pollWait(0xffffad272448, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1d98, 0x72, 0x0, 0x1, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f1d80, 0x4004e526d1, 0x1, 0x1, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f1d80, 0x4004e526d1, 0x1, 0x1, 0x1, 0x0, 0x4004d4c4e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400018a6d8, 0x4004e526d1, 0x1, 0x1, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*connReader).backgroundRead(0x4004e526c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:692 +0x50 created by net/http.(*connReader).startBackgroundRead /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:688 +0xc4 goroutine 201 [IO wait]: internal/poll.runtime_pollWait(0xffffad272278, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1f98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f1f80, 0x4004e65000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f1f80, 0x4004e65000, 0x1000, 0x1000, 0x0, 0x0, 0x4002566c78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400018a6f8, 0x4004e65000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004cb30e0, 0x4004e65000, 0x1000, 0x1000, 0x60, 0x0, 0x12be201) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004e76060) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004e76060, 0x1, 0x4004e1ef00, 0x43d584, 0x0, 0x0, 0x4002566d68) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004cb30e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 202 [select]: net/http.(*persistConn).writeLoop(0x4004cb30e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc Initializing write barrier = 8000 2021-08-16T13:51:07.189+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:51:07.190+00:00 [Info] Setting ipv6=false 2021-08-16T13:51:07.200+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:51:07.202+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:51:07.202+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:51:07.202+00:00 [Info] Setting log level to Info 2021-08-16T13:51:07.202+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:51:07.205+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:51:07 audit: created new audit service 2021-08-16T13:51:07.213+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:51:07.213+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:51:07.214+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:51:07.214+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:51:07.214+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:51:07.224+00:00 [Info] security context initialized 2021-08-16T13:51:07.224+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:51:07.224+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:51:07.224+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:51:07.224+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:51:07.224+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:51:07.237+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:51:07.250+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:51:07.251+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:51:07.265+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:51:07.267+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:51:07.267+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:51:07.267+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:51:07.268+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:51:07.268+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:51:07.269+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:51:07.271+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:51:07.322+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:51:07.322+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 53 us 2021-08-16T13:51:07.323+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:51:07.324+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:51:07.325+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:51:07.329+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:51:07.329+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:51:07.329+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:51:07.329+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:51:07.329+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:51:07.329+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:07.329+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:07.330+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:51:07.330+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:51:07.330+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:51:07.330+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:07.330+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:51:07.330+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:51:07.330+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:51:07.330+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:51:07.330+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:51:07.330+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:51:07.330+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:51:07.337+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 6.772954ms 2021-08-16T13:51:07.337+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:51:07.338+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:51:07.338+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:07.338+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:51:07.338+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:51:07.338+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:51:07.338+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:51:07.351+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:51:07.351+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:51:07.352+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:51:07.352+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:51:07.352+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x400247db1f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x400247db1f pc=0xbd8ea4] goroutine 185 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004d069f0 sp=0x4004d069c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004d06a30 sp=0x4004d069f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x40001a1d40, 0x4004d38600, 0x11a9890, 0x4004d06b40, 0x40001a1d58, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004d06ab0 sp=0x4004d06a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x40001a1d40, 0x4004d38600, 0x11a9890, 0x4004d06b40, 0x40001a1d58, 0xffff8a237758, 0x78, 0xffff8a2b5a90) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004d06af0 sp=0x4004d06ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4000032b40, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004d06da0 sp=0x4004d06af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4000032b40, 0x40002bd5e0, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004d06e60 sp=0x4004d06da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x40046a9690, 0x40046a96a0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004d07200 sp=0x4004d06e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x40046a9690, 0x40046a96a0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004d078f0 sp=0x4004d07200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x40046f5f70, 0x4004d68380, 0x40002d5600, 0x40046a9670) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004d07fb0 sp=0x4004d078f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004d07fb0 sp=0x4004d07fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x4000096000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x4000283710, 0x4, 0x1136a87, 0xf, 0x4000211e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 34 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x40002a80d8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 35 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x40002a80f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 36 [IO wait]: internal/poll.runtime_pollWait(0xffff6057d1f8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400028e798, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400028e780, 0x40002fd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400028e780, 0x40002fd000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400028c098, 0x40002fd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x4000280780, 0x4000216001, 0xdff, 0xdff, 0x40001998e8, 0x6902d8, 0x400020d948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x40002a8138, 0x4000216001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40001e6840, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40001e6840, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40001e6840, 0xf5a960, 0x40001ba500, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x40001ba4e0, 0x40001c22c0, 0x400020db28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x4000180190, 0x12f8f98, 0x40001ba4e0, 0x40001b4c57, 0x16, 0x40001c6fc0, 0x4000199c01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x4000180190, 0x12f8f98, 0x40001ba4e0, 0x4000180190, 0x40001b4468, 0x40001b4470, 0x4000208080, 0x40001c21e0, 0xee0980, 0x40001c6fc0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x4000180190, 0x12f8f98, 0x40001ba4e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4000282510, 0x4000199f78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x4000093f78, 0x4000282510, 0x12db8a8, 0x40002a8108, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x4000282510, 0x400029e460, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x4000282510, 0x400029e460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 37 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 21 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x4000190200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 38 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 39 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 40 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 41 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 42 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40002c77a0, 0x4, 0x40002c7668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40002c77a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x40001ba000, 0x40002c77a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x40001b8000, 0x40002c77a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 9 [IO wait]: internal/poll.runtime_pollWait(0xffff6057c8e8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000134798, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000134780, 0x40044aa000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000134780, 0x40044aa000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000126208, 0x40044aa000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000d200, 0x40044aa000, 0x1000, 0x1000, 0x4000186d80, 0x200000003, 0x4000186d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000125980) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000125980, 0x400001800a, 0x40044bb8c8, 0x4d3b10, 0x40001ba060, 0x4000018000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000125980, 0x0, 0x1, 0x0, 0x40001ba078, 0x4000018000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000220a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000220a20, 0x4000255000, 0x1000, 0x1000, 0x40044bb9b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40001d9840, 0x4000255000, 0x1000, 0x1000, 0x4000280298, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40001d9840, 0x4000255000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001d9880, 0x4000255000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40044bbdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40044bbdd0, 0xa, 0x4000255201, 0x1, 0xd1f, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40044bbdd0, 0x400025520a, 0x1, 0x40001b5e0e, 0x0, 0x40002552e1, 0x1, 0xd1f, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40044bbdd0, 0x400445440a, 0x40001b5e0e, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40044542b0, 0x400003a0d8, 0x17, 0x40044bbf20, 0x40044a1e60, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x40044542b0, 0x113075a, 0x7, 0x40044a1e60, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x4004454270) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 7 [select]: net/http.(*persistConn).readLoop(0x40001c6480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 8 [select]: net/http.(*persistConn).writeLoop(0x40001c6480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 50 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40002847c0, 0x12c18bc, 0x1, 0x400008de58, 0x4000038720, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x40002c06c8, 0x4000038720, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400008dfa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4000080040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 51 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4000080040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 32 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x40004fc380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 25 [IO wait]: internal/poll.runtime_pollWait(0xffff6057d028, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000134118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000134100, 0x4000230000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000134100, 0x4000230000, 0x1000, 0x1000, 0x4000156be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010008, 0x4000230000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c480, 0x4000230000, 0x1000, 0x1000, 0x4000156cf8, 0x67867c, 0x40002440c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40000820c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40000820c0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000c480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 54 [IO wait]: internal/poll.runtime_pollWait(0xffff6057d110, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000208218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000208200, 0x4000217000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000208200, 0x4000217000, 0x1000, 0x1000, 0x400015aa88, 0x68f7a4, 0x400029a730) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b80e0, 0x4000217000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c6480, 0x4000217000, 0x1000, 0x1000, 0xf12180, 0x4000294270, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001bac00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001bac00, 0xa, 0x1008880, 0x4000294230, 0x199, 0x1000, 0x400015ab00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40001bac00, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400011e6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400011e6c0, 0x400015c000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400012c180, 0x400015c000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400012c180, 0x400015c000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400012c1c0, 0x400015c000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x400029a6e0, 0x0, 0x400015aeb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400029a6e0, 0x0, 0x0, 0x400015af70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400029a6e0, 0xee0d40, 0x4000294230, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40002b95f0, 0x4000124420, 0x4000038a20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 26 [select]: net/http.(*persistConn).writeLoop(0x400000c480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 67 [select]: net/http.(*persistConn).readLoop(0x40002f46c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 116 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40002865c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 44 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400012c700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 31 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x40004fc380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 138 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40004fc8c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 33 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x40004fc380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 46 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x40004fc380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 68 [select]: net/http.(*persistConn).writeLoop(0x40002f46c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 119 [IO wait]: internal/poll.runtime_pollWait(0xffff6057c548, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400028fa98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400028fa80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400028fa80, 0x58ec48, 0x4000183380, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40047633e0, 0x4714f4, 0x4000183380, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40047633e0, 0x40001b5e00, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x40047684c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 61 [select]: net/http.(*persistConn).readLoop(0x400000d200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 62 [select]: net/http.(*persistConn).writeLoop(0x400000d200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 10 [IO wait]: internal/poll.runtime_pollWait(0xffff6057cab8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400028e118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400028e100, 0x4002516000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400028e100, 0x4002516000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010040, 0x4002516000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c240, 0x4002516000, 0x1000, 0x1000, 0x4000187200, 0x200000003, 0x4000187200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40004771a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40004771a0, 0x400001800a, 0x40044b7858, 0x4344e0, 0x40001ba060, 0x4000018000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40004771a0, 0x41c330, 0xffff6056c2a0, 0x0, 0x0, 0x4000018000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4002534b10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4002534b10, 0x40000cd000, 0x1000, 0x1000, 0x40044b79b8, 0x0, 0x40044b7988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000286140, 0x40000cd000, 0x1000, 0x1000, 0x40002be480, 0x3, 0x40044b7be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000286140, 0x40000cd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000286380, 0x40000cd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40044b7dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40044b7dd0, 0xa, 0x40000cd201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40044b7dd0, 0x40000cd20a, 0x1, 0x40001b5e0b, 0x0, 0x40000cd263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40044b7dd0, 0x40001c240a, 0x40001b5e0b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40044542b0, 0x4000042150, 0x24, 0x40044b7f20, 0x40024421c8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x40044542b0, 0x113075a, 0x7, 0x40024421c8, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x4004454270) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 117 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000287040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 13 [select]: net/http.(*persistConn).readLoop(0x400000c240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 14 [select]: net/http.(*persistConn).writeLoop(0x400000c240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 82 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000287080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 83 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x4000124a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 84 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x4000220b70) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 85 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x4000208280) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 86 [IO wait]: internal/poll.runtime_pollWait(0xffff6057c800, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000208398, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000208380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000208380, 0x680da0, 0x40002c4698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x400000f650, 0x40002447e0, 0x4000082200, 0x4000001708) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x400000f650, 0x12ed9d0, 0x400000f650, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x4000208300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 87 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40001fc8f0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 88 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40001fc8f0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 89 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x40001fc8f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 90 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x4000221650) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 96 [IO wait]: internal/poll.runtime_pollWait(0xffff6057c630, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000208698, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000208680, 0x40000d3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000208680, 0x40000d3000, 0x1000, 0x1000, 0x40044b5688, 0x69f178, 0x40044b56f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010060, 0x40000d3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c6a20, 0x40000d3000, 0x1000, 0x1000, 0x5, 0x40023da500, 0x40044b57c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000477560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000477560, 0x68f90a, 0x4000400f20, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000477560, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40044a8480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40044a8480, 0x40044c0000, 0x1000, 0x1000, 0x40044b5998, 0x44e884, 0x40044b5948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400012c380, 0x40044c0000, 0x1000, 0x1000, 0x40001bb318, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400012c380, 0x40044c0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400012c3c0, 0x40044c0000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40044b5d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40044b5d80, 0xa, 0x40044c0201, 0x1, 0xffff605b79f8, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40044b5d80, 0x40044c020a, 0x1, 0x40024e5200, 0x0, 0x40044c0259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40044b5d80, 0x400256680a, 0x40024e5200, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40044542b0, 0x40024443e0, 0x1f, 0x40044b5ee0, 0x400000f938, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x40044542b0, 0x113075a, 0x7, 0x4002562620, 0xd, 0x400000f938, 0x0, 0x0, 0x2) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x4004454270, 0x4002562620, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 94 [IO wait]: internal/poll.runtime_pollWait(0xffff6057c718, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000208518, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000208500, 0x400446d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000208500, 0x400446d000, 0x1000, 0x1000, 0x40044acbe0, 0x469b10, 0x400447a600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b8568, 0x400446d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c67e0, 0x400446d000, 0x1000, 0x1000, 0x40044accf8, 0x67867c, 0x4000183080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000125f20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000125f20, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40001c67e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 93 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40001f80e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 95 [select]: net/http.(*persistConn).writeLoop(0x40001c67e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 120 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x40024f6b40, 0x40000cae00, 0x40000cae70, 0x40046a89b0, 0x12e8f90, 0x4004562540, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x4004729754, 0x5, 0x40047684c0, 0x12eca70, 0x4004768340, 0x130d478, 0x400045a690, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40045628c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40045628c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 114 [select]: net/http.(*persistConn).readLoop(0x40001c6a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 115 [select]: net/http.(*persistConn).writeLoop(0x40001c6a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 118 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000287bc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 101 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x40001bdcc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 102 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x40001bdcc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 103 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x40024f6b40, 0x40046a89b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 121 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x4004562540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 122 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x4004562540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 123 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x4002512cf0, 0x4004774540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 124 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x4000286580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 125 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x40004a1220) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 126 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x40004a1220) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 127 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x40000105e8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 128 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x40004a1220, 0x40047745a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 129 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40002847c0, 0x1144559, 0x1b, 0x4004cc9e38, 0x40047746c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004cbc350, 0x40047746c0, 0xf1f700, 0x40004fa040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004cc9f80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004639ec0, 0x4004cbc350) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 146 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 147 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x4000266d20, 0xd00007, 0x4004ca1218, 0x5, 0x4004cbc360, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 148 [IO wait]: internal/poll.runtime_pollWait(0xffff6057c460, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400028fc98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400028fc80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400028fc80, 0xffff8a2b5108, 0x30, 0x4004771bf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004763980, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004763980, 0x4004771bf0, 0x11ae500, 0x4004562a20, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40045629a0, 0x12ed9d0, 0x4004763980, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40045629a0, 0x12ed9d0, 0x4004763980, 0x4000096000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 149 [IO wait]: internal/poll.runtime_pollWait(0xffff6057c378, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400028fd98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400028fd80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400028fd80, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004763998, 0xffff8a2b5108, 0xffff60084c50, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004763998, 0x665e1c, 0x0, 0xfe5e60, 0x400256e5c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x40047639b0, 0x4004771c80, 0x11ae500, 0x4004562b00, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x4004562a80, 0x12ec710, 0x40047639b0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x4004562a80, 0x12ec710, 0x40047639b0, 0x4000096000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 139 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40004fc8c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 110 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400012d1c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 105 [IO wait]: internal/poll.runtime_pollWait(0xffff600d34c0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000209718, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000209700, 0x40046c1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000209700, 0x40046c1000, 0x1000, 0x1000, 0x4004d0cbe0, 0x469b10, 0x4004cbb080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b8780, 0x40046c1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004cb07e0, 0x40046c1000, 0x1000, 0x1000, 0x4004d0ccf8, 0x67867c, 0x40046b09c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400447f740) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x400447f740, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004cb07e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 106 [select]: net/http.(*persistConn).writeLoop(0x4004cb07e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 111 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x4000401c30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 137 [semacquire]: sync.runtime_Semacquire(0x40046f5f78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x40046f5f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x40002d5600, 0x400011e3f0, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xfffffbc5d969, 0x29, 0xfffffbc5d9cb, 0x25, 0x40046b1aa0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004ca04f0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xfffffbc5d969, 0x29, 0xfffffbc5d9cb, 0x25, 0x40046b1aa0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004ca04f0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004d07190, 0x4004d06e60, 0x400011e3f0, 0x40001cb8c0, 0x400012c700, 0x0, 0x1bf, 0x4004474da8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x4000096000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004ca04f0, 0xb, 0x4004ca04e8, 0x6, 0x4004ca0500, 0xd, 0x4004761260, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x4000096000, 0x4000470000, 0x4000094030) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x4000096000, 0x4000226840, 0x2, 0x2, 0x4000226850, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 155 [IO wait]: internal/poll.runtime_pollWait(0xffff6057c9d0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40004eeb98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40004eeb80, 0x4002568000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40004eeb80, 0x4002568000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000126050, 0x4002568000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40002f46c0, 0x4002568000, 0x1000, 0x1000, 0x0, 0x0, 0x4000066b28) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40000824e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000824e0, 0xa, 0x0, 0x0, 0x0, 0x0, 0x400256e418) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40000824e0, 0x412bc0, 0x0, 0x0, 0x400256e4c8, 0x412bc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004ccc510) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004ccc510, 0x4004767000, 0x200, 0x200, 0x4004771c50, 0x30, 0x400256e578) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004768b00, 0x4004767000, 0x200, 0x200, 0x0, 0xffff8a2b5108, 0x400256e568) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004768b00, 0x4004767000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004768b40, 0x4004767000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40047731e0, 0x12ec700, 0x400256e6b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40047731e0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40047731e0, 0xee0d40, 0x40002672c0, 0x12db068, 0x4004768b40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400476f320, 0x4004cc0a20, 0x4004774c60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 157 [IO wait]: internal/poll.runtime_pollWait(0xffff600d33d8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40004ef398, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40004ef380, 0x4004dbb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40004ef380, 0x4004dbb000, 0x1000, 0x1000, 0x40000dcad8, 0x412bc0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400028c440, 0x4004dbb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004dd6240, 0x4004dbb000, 0x1000, 0x1000, 0x4004cc0ba0, 0x60, 0x58) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004dd4ea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004dd4ea0, 0xa, 0x0, 0x0, 0x0, 0x0, 0x400256e418) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004dd4ea0, 0x412bc0, 0x0, 0x0, 0x400256e4c8, 0x412bc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004ccc750) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004ccc750, 0x4004767400, 0x200, 0x200, 0x4004771c50, 0x30, 0x400256e578) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004768c40, 0x4004767400, 0x200, 0x200, 0x0, 0xffff8a2b5108, 0x400256e568) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004768c40, 0x4004767400, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004768c80, 0x4004767400, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004773760, 0x12ec700, 0x400256e6b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004773760, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004773760, 0xee0d40, 0x40002673b0, 0x12db068, 0x4004768c80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400476f4d0, 0x4004cc0c60, 0x4004774ea0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 140 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40004fc8c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 141 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40004fc8c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 142 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x400012d180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 170 [IO wait]: internal/poll.runtime_pollWait(0xffff600d35a8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40004ef298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40004ef280, 0x4004ded000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40004ef280, 0x4004ded000, 0x1000, 0x1000, 0x4004d0bbe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400028c448, 0x4004ded000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004dd6120, 0x4004ded000, 0x1000, 0x1000, 0x4004d0bcf8, 0x67867c, 0x40044f1860) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004dd4fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004dd4fc0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004dd6120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 145 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4000401d90, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 162 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4000401d90, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 163 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4000401d90) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 164 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40004fc8c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 165 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40002847c0, 0x1144559, 0x1b, 0x4004cc5e38, 0x40044f0b40, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4000123a80, 0x40044f0b40, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004cc5f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004dd41e0, 0x4000123a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 171 [select]: net/http.(*persistConn).writeLoop(0x4004dd6120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 168 [select]: net/http.(*persistConn).readLoop(0x4004dd6240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 169 [select]: net/http.(*persistConn).writeLoop(0x4004dd6240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 112 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x4000401c30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 113 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x4000401c30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 178 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x4000401c30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 179 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004685a00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 180 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x4004685a00, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 181 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x4000096000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 182 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x4000096000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 183 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x4000096000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 184 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:51:12.522+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:51:12.522+00:00 [Info] Setting ipv6=false 2021-08-16T13:51:12.532+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:51:12.539+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:51:12.539+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:51:12.539+00:00 [Info] Setting log level to Info 2021-08-16T13:51:12.539+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:51:12.541+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:51:12 audit: created new audit service 2021-08-16T13:51:12.549+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:51:12.550+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:51:12.550+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:51:12.550+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:51:12.550+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:51:12.561+00:00 [Info] security context initialized 2021-08-16T13:51:12.561+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:51:12.561+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:51:12.561+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:51:12.561+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:51:12.561+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:51:12.575+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:51:12.575+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:51:12.579+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:51:12.603+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:51:12.604+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:51:12.604+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:51:12.605+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:51:12.605+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:51:12.606+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:51:12.608+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:51:12.609+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:51:12.658+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:51:12.659+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 51 us 2021-08-16T13:51:12.659+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:51:12.661+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:51:12.661+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:51:12.665+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:51:12.665+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:51:12.665+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:51:12.665+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:51:12.665+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:51:12.665+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:12.665+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:12.666+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:51:12.667+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:51:12.667+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:51:12.667+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:12.667+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:51:12.667+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:51:12.667+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:51:12.667+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:51:12.667+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:51:12.667+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:51:12.667+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:51:12.673+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 5.19014ms 2021-08-16T13:51:12.673+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:51:12.674+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:51:12.674+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:12.674+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:51:12.674+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:51:12.674+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:51:12.674+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:51:12.686+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:51:12.686+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:51:12.686+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:51:12.686+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:51:12.686+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x400467a6df fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x400467a6df pc=0xbd8ea4] goroutine 178 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x40023d29f0 sp=0x40023d29c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x40023d2a30 sp=0x40023d29f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x400467eb60, 0x40045bf800, 0x11a9890, 0x40023d2b40, 0x400467eb78, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x40023d2ab0 sp=0x40023d2a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x400467eb60, 0x40045bf800, 0x11a9890, 0x40023d2b40, 0x400467eb78, 0xc18f8c, 0x40023d2c18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x40023d2af0 sp=0x40023d2ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4004490640, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x40023d2da0 sp=0x40023d2af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4004490640, 0x400439d110, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x40023d2e60 sp=0x40023d2da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004e4c290, 0x4004e4c2a0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x40023d3200 sp=0x40023d2e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004e4c290, 0x4004e4c2a0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x40023d38f0 sp=0x40023d3200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004e2b330, 0x4004e05500, 0x4004d19080, 0x4004e4c270) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x40023d3fb0 sp=0x40023d38f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x40023d3fb0 sp=0x40023d3fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x4000344380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x40000fb6e0, 0x4, 0x1136a87, 0xf, 0x40000f9e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 7 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400000e690) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 8 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400000e6a8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 9 [IO wait]: internal/poll.runtime_pollWait(0xffff70884198, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0898, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0880, 0x40001a3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0880, 0x40001a3000, 0x1000, 0x1000, 0x1311860, 0x47b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000105b0, 0x40001a3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x4000082cc0, 0x4000308001, 0xdff, 0xdff, 0x400009f8e8, 0x6902d8, 0x40000f5948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400000e6f0, 0x4000308001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40000ce9a0, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40000ce9a0, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40000ce9a0, 0xf5a960, 0x4000082da0, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4000082d80, 0x400031a0c0, 0x40000f5b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400008c820, 0x12f8f98, 0x4000082d80, 0x40003007d7, 0x16, 0x400030e120, 0x400009fc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400008c820, 0x12f8f98, 0x4000082d80, 0x400008c820, 0x4000036c58, 0x4000036c60, 0x40000f0900, 0x40000466e0, 0xee0980, 0x400030e120, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400008c820, 0x12f8f98, 0x4000082d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x40000fa3c0, 0x400009ff78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400009df78, 0x40000fa3c0, 0x12db8a8, 0x400000e6c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x40000fa3c0, 0x40000b12c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x40000fa3c0, 0x40000b12c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 10 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 11 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 12 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 13 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 14 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 34 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400010c200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 15 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40003fbfa0, 0x4, 0x40003fbe68, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40003fbfa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x40003fbfa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x40003fbfa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 39 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001944c0, 0x12c18bc, 0x1, 0x4000439e58, 0x40003506c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x40003faec8, 0x40003506c0, 0x40003faec8, 0x679704) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x400011a008) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000439fa0, 0x4714f4, 0x3b9aca00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x400032a100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 37 [select]: net/http.(*persistConn).readLoop(0x400039e120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 38 [select]: net/http.(*persistConn).writeLoop(0x400039e120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 40 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x400032a100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 69 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x400240e380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 43 [IO wait]: internal/poll.runtime_pollWait(0xffff70883fc8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0280, 0x400014a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0280, 0x400014a000, 0x1000, 0x1000, 0x4000066be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000103e8, 0x400014a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400039ec60, 0x400014a000, 0x1000, 0x1000, 0x4000066cf8, 0x67867c, 0x4000038a80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003b1a40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40003b1a40, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400039ec60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 44 [select]: net/http.(*persistConn).writeLoop(0x400039ec60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 22 [IO wait]: internal/poll.runtime_pollWait(0xffff708840b0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0180, 0x40001a2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0180, 0x40001a2000, 0x1000, 0x1000, 0x400001fa88, 0x68f7a4, 0x40000121b0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000100d0, 0x40001a2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400039e120, 0x40001a2000, 0x1000, 0x1000, 0xf12180, 0x40003460e0, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003b1500) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40003b1500, 0xa, 0x1008880, 0x40003460a0, 0x199, 0xf03, 0x400001fb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40003b1500, 0x2, 0x0, 0x9, 0x9b5, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000096d50) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000096d50, 0x40000f6000, 0xe00, 0xe00, 0x45c, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000080000, 0x40000f6000, 0xe00, 0xe00, 0x400, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000080000, 0x40000f6000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000080140, 0x40000f6000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4000012160, 0x0, 0x400001feb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4000012160, 0x0, 0x0, 0x400001ff70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4000012160, 0xee0d40, 0x40003460a0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4000372000, 0x40001163c0, 0x4000016000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 82 [select]: net/http.(*persistConn).readLoop(0x4002460480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 25 [select]: net/http.(*persistConn).readLoop(0x400030eea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 26 [select]: net/http.(*persistConn).writeLoop(0x400030eea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 50 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400235e580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 114 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400235e680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 52 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x400240e380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 87 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x400032b0c0, 0x4000440930, 0x40004409a0, 0x4000317e10, 0x12e8f90, 0x4004422700, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x4004c127f4, 0x5, 0x400032afc0, 0x12eca70, 0x400235fd80, 0x130d478, 0x40001f4850, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x4004422c40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x4004422c40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 86 [IO wait]: internal/poll.runtime_pollWait(0xffff708834e8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1698, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f1680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f1680, 0x12d9d48, 0x400032a5c0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x400239f5f0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x400239f5f0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x400032afc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 63 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400439a7c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 27 [IO wait]: internal/poll.runtime_pollWait(0xffff708836b8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f1100, 0x400011c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f1100, 0x400011c000, 0x1000, 0x1000, 0x400009b688, 0x69f178, 0x400009b6f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000314100, 0x400011c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40024605a0, 0x400011c000, 0x1000, 0x1000, 0x5, 0x4004410f00, 0x400009b7c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400033aa20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400033aa20, 0x68f90a, 0x40003226e0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400033aa20, 0x412ff8, 0x400033b1a0, 0x60, 0x58, 0x10786a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40003f3920) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40003f3920, 0x40024b9000, 0x1000, 0x1000, 0x400009b998, 0x44e884, 0x10786a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400235e380, 0x40024b9000, 0x1000, 0x1000, 0x4000164898, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400235e380, 0x40024b9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400235e3c0, 0x40024b9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400009bd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400009bd80, 0xa, 0x40024b9201, 0x1, 0xffff70683ef8, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400009bd80, 0x40024b920a, 0x1, 0x40023e5db0, 0x0, 0x40024b9259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400009bd80, 0x400246bc0a, 0x40023e5db0, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000e5630, 0x400245c2a0, 0x1f, 0x400009bee0, 0x40003124f8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x40000e5630, 0x113075a, 0x7, 0x4002458590, 0xd, 0x40003124f8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x40000e55f0, 0x4002458590, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 70 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x400240e380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 71 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x400240e380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 72 [IO wait]: internal/poll.runtime_pollWait(0xffff70883888, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000468118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000468100, 0x40044e3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000468100, 0x40044e3000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010410, 0x40044e3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400039ea20, 0x40044e3000, 0x1000, 0x1000, 0x400033ca80, 0x200000003, 0x400033ca80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40023ec780) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40023ec780, 0x400014e00a, 0x40000a18c8, 0x4d3b10, 0x40000820c0, 0x400014e000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40023ec780, 0x0, 0x1, 0x0, 0x40000820d8, 0x400014e000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400248e870) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400248e870, 0x4000152000, 0x1000, 0x1000, 0x40000a19b8, 0x0, 0x40000a1988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400032a240, 0x4000152000, 0x1000, 0x1000, 0x4000102a80, 0x3, 0x40000a1be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400032a240, 0x4000152000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400032a400, 0x4000152000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40000a1dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000a1dd0, 0xa, 0x4000152201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40000a1dd0, 0x400015220a, 0x1, 0x4002458579, 0x0, 0x40001522e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40000a1dd0, 0x400248a20a, 0x4002458579, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000e5630, 0x400037e7e0, 0x17, 0x40000a1f20, 0x400042e408, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x40000e5630, 0x113075a, 0x7, 0x400042e408, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x40000e55f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 73 [IO wait]: internal/poll.runtime_pollWait(0xffff70883a58, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000342b18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000342b00, 0x4002368000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000342b00, 0x4002368000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000126278, 0x4002368000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400030eea0, 0x4002368000, 0x1000, 0x1000, 0x400033cc00, 0x200000003, 0x400033cc00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001172c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001172c0, 0x400014e00a, 0x40004358c8, 0x4d3b10, 0x40000820c0, 0x400014e000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40001172c0, 0x0, 0x1, 0x0, 0x40000820d8, 0x400014e000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40003f38c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40003f38c0, 0x40024b8000, 0x1000, 0x1000, 0x40004359b8, 0x44d8a8, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400235e300, 0x40024b8000, 0x1000, 0x1000, 0x4000164898, 0x1, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400235e300, 0x40024b8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400235e340, 0x40024b8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000435dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000435dd0, 0xa, 0x40024b8201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000435dd0, 0x40024b820a, 0x1, 0x400245898b, 0x0, 0x40024b8263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000435dd0, 0x400012820a, 0x400245898b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000e5630, 0x400037a4e0, 0x24, 0x4000435f20, 0x40044df3f8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x40000e5630, 0x113075a, 0x7, 0x40044df3f8, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x40000e55f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 158 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x4004422e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 55 [select]: net/http.(*persistConn).readLoop(0x400039ea20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 56 [select]: net/http.(*persistConn).writeLoop(0x400039ea20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 83 [select]: net/http.(*persistConn).writeLoop(0x4002460480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 32 [select]: net/http.(*persistConn).readLoop(0x40024605a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 33 [select]: net/http.(*persistConn).writeLoop(0x40024605a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 74 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400032ab80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 75 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x4000164a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 76 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x40023b4ba0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 77 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x4000342180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 78 [IO wait]: internal/poll.runtime_pollWait(0xffff708835d0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000342398, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000342380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000342380, 0x0, 0x4000054e98, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40044df698, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40044df698, 0x12ed9d0, 0x40044df698, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x4000342280) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 79 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40000e5a00, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 80 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40000e5a00, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 81 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x40000e5a00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 98 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x40023b5680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 88 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x40000b1ae0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 64 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400235f540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 101 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x400033e000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 156 [semacquire]: sync.runtime_Semacquire(0x4004e2b338) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004e2b330) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x4004d19080, 0x40003f2b10, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffea6ab969, 0x29, 0xffffea6ab9cb, 0x25, 0x4004d0d7a0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x400463f570, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffea6ab969, 0x29, 0xffffea6ab9cb, 0x25, 0x4004d0d7a0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x400463f570, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x40023d3190, 0x40023d2e60, 0x40003f2b10, 0x4000311680, 0x400235e580, 0x0, 0xffff703ecd50, 0x78, 0xffff99156f40, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x4000344380, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x400463f570, 0xb, 0x400463f568, 0x6, 0x400463f580, 0xd, 0x4004636ae0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x4000344380, 0x40003b0000, 0x400030a720) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x4000344380, 0x4004476b60, 0x2, 0x2, 0x4004476b70, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 134 [IO wait]: internal/poll.runtime_pollWait(0xffff703f33d8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000469c98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000469c80, 0x4004c5f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000469c80, 0x4004c5f000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010968, 0x4004c5f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004be8480, 0x4004c5f000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40047d1a40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40047d1a40, 0xa, 0x0, 0x0, 0x0, 0x0, 0x4004609418) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40047d1a40, 0x412bc0, 0x0, 0x0, 0x40046094c8, 0x412bc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004d48a80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004d48a80, 0x4004d7a200, 0x200, 0x200, 0x4004641800, 0x30, 0x4004609578) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400439bc40, 0x4004d7a200, 0x200, 0x200, 0x0, 0xffff99156f18, 0x4004609568) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400439bc40, 0x4004d7a200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400439bc80, 0x4004d7a200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004d77080, 0x12ec700, 0x40046096b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004d77080, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004d77080, 0xee0d40, 0x40043bf180, 0x12db068, 0x400439bc80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004645cb0, 0x4004679560, 0x4004d0cea0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 106 [IO wait]: internal/poll.runtime_pollWait(0xffff70883970, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000342e18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000342e00, 0x4002419000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000342e00, 0x4002419000, 0x1000, 0x1000, 0x400006bbe0, 0x469b10, 0x400033de00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001262b8, 0x4002419000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400030f560, 0x4002419000, 0x1000, 0x1000, 0x400006bcf8, 0x67867c, 0x40043b9c20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40043a0840) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40043a0840, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400030f560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 107 [select]: net/http.(*persistConn).writeLoop(0x400030f560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 89 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x40000b1ae0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 90 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x400032b0c0, 0x4000317e10) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 115 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x4004422700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 116 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x4004422700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 117 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x40023cad80, 0x40044ebd40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 118 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x400235e640) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 119 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x40043c2be0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 120 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x40043c2be0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 121 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x40043a83a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 122 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x40043c2be0, 0x40044ebda0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 123 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001944c0, 0x1144559, 0x1b, 0x4004571e38, 0x40044ebec0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004660e30, 0x40044ebec0, 0xf1f700, 0x4000195d40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004571f80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40046784e0, 0x4004660e30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 124 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 125 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x40043be9b0, 0xd00007, 0x4004672238, 0x5, 0x4004660e40, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 126 [IO wait]: internal/poll.runtime_pollWait(0xffff70883400, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40043de998, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40043de980, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40043de980, 0xffff99156f18, 0x30, 0x40046417a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004676120, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004676120, 0x40046417a0, 0x11ae500, 0x40045b8400, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40045b8380, 0x12ed9d0, 0x4004676120, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40045b8380, 0x12ed9d0, 0x4004676120, 0x4000344380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 127 [IO wait]: internal/poll.runtime_pollWait(0xffff70883318, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40043dea98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40043dea80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40043dea80, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004676138, 0xffff99156f18, 0xffff70338e68, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004676138, 0x665e1c, 0x0, 0xfe5e60, 0x40046095c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004676150, 0x4004641830, 0x11ae500, 0x40045b84e0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40045b8460, 0x12ec710, 0x4004676150, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40045b8460, 0x12ec710, 0x4004676150, 0x4000344380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 153 [IO wait]: internal/poll.runtime_pollWait(0xffff708837a0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1018, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f1000, 0x4000132000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f1000, 0x4000132000, 0x1000, 0x1000, 0x1, 0x40047972f0, 0x400015ca78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000314110, 0x4000132000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4002460480, 0x4000132000, 0x1000, 0x1000, 0x400235ff00, 0x0, 0x105f560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400033ab40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400033ab40, 0x100a, 0x4004380123, 0x11320c7, 0x9, 0x597c01, 0xe4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400033ab40, 0xd4fa28, 0x400044c080, 0x46, 0x40003f74a8, 0x40b2f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004797710) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004797710, 0x40047d3200, 0x200, 0x200, 0x60, 0x0, 0x4000043401) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004c58000, 0x40047d3200, 0x200, 0x200, 0x4004c38a20, 0x43d584, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004c58000, 0x40047d3200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004c58040, 0x40047d3200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004beb760, 0x680d00, 0x40003f76b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004beb760, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004beb760, 0xee0d40, 0x400008d040, 0x12db068, 0x4004c58040) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004c305a0, 0x40047d0c00, 0x4004c38a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 157 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x4004422e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 146 [IO wait]: internal/poll.runtime_pollWait(0xffff703f34c0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000469798, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000469780, 0x4004c37000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000469780, 0x4004c37000, 0x1000, 0x1000, 0x400015dbe0, 0x469b10, 0x4004675800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010810, 0x4004c37000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40043b39e0, 0x4004c37000, 0x1000, 0x1000, 0x400015dcf8, 0x67867c, 0x4004c382a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40047d06c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40047d06c0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40043b39e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 147 [select]: net/http.(*persistConn).writeLoop(0x40043b39e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 135 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4004c584c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 159 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x4004422e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 160 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x4004422e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 161 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4004c58480) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 173 [IO wait]: internal/poll.runtime_pollWait(0xffff703f35a8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000469b18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000469b00, 0x4004da1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000469b00, 0x4004da1000, 0x1000, 0x1000, 0x4004d10be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010970, 0x4004da1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004be8360, 0x4004da1000, 0x1000, 0x1000, 0x4004d10cf8, 0x67867c, 0x4004da4060) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40047d1b60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40047d1b60, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004be8360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 164 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004685810, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 165 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004685810, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 166 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4004685810) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 167 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x4004422e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 168 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001944c0, 0x1144559, 0x1b, 0x400009fe38, 0x4004c39320, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004789a30, 0x4004c39320, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400009ff80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40047d0fc0, 0x4004789a30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 174 [select]: net/http.(*persistConn).writeLoop(0x4004be8360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 171 [select]: net/http.(*persistConn).readLoop(0x4004be8480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 172 [select]: net/http.(*persistConn).writeLoop(0x4004be8480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 136 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x40046856b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 137 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x40046856b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 138 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x40046856b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 139 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x40046856b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 140 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004d0b400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 141 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x4004d0b400, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 142 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x4000344380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 143 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x4000344380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 144 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x4000344380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 145 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:51:17.839+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:51:17.839+00:00 [Info] Setting ipv6=false 2021-08-16T13:51:17.845+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:51:17.849+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:51:17.849+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:51:17.849+00:00 [Info] Setting log level to Info 2021-08-16T13:51:17.849+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:51:17.852+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:51:17 audit: created new audit service 2021-08-16T13:51:17.860+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:51:17.860+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:51:17.860+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:51:17.860+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:51:17.860+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:51:17.872+00:00 [Info] security context initialized 2021-08-16T13:51:17.872+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:51:17.872+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:51:17.872+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:51:17.872+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:51:17.872+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:51:17.885+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:51:17.886+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:51:17.890+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:51:17.911+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:51:17.911+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:51:17.912+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:51:17.912+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:51:17.913+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:51:17.915+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:51:17.915+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:51:17.916+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:51:17.965+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:51:17.966+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 53 us 2021-08-16T13:51:17.966+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:51:17.967+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:51:17.968+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:51:17.972+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:51:17.972+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:51:17.972+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:51:17.972+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:51:17.972+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:51:17.972+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:17.972+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:17.976+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:51:17.976+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:51:17.976+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:51:17.977+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:17.977+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:51:17.977+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:51:17.977+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:51:17.977+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:51:17.977+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:51:17.977+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:51:17.977+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:51:17.982+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 5.056086ms 2021-08-16T13:51:17.983+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:51:17.984+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:51:17.984+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:17.984+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:51:17.984+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:51:17.984+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:51:17.984+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:51:17.995+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:51:17.995+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:51:17.995+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:51:17.995+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:51:17.995+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x4004770d9f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x4004770d9f pc=0xbd8ea4] goroutine 143 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004e8e9f0 sp=0x4004e8e9c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004e8ea30 sp=0x4004e8e9f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x4004e9a000, 0x4004710c00, 0x11a9890, 0x4004e8eb40, 0x4004e9a018, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004e8eab0 sp=0x4004e8ea40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x4004e9a000, 0x4004710c00, 0x11a9890, 0x4004e8eb40, 0x4004e9a018, 0x3f, 0x4004e8eb78, 0x412ddc) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004e8eaf0 sp=0x4004e8eab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x40044fe780, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004e8eda0 sp=0x4004e8eaf0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x40044fe780, 0x4000465d50, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004e8ee60 sp=0x4004e8eda0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x400475b5a0, 0x400475b5b0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004e8f200 sp=0x4004e8ee60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x400475b5a0, 0x400475b5b0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004e8f8f0 sp=0x4004e8f200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004dc1ae0, 0x4004db9180, 0x40044fb600, 0x400475b580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004e8ffb0 sp=0x4004e8f8f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004e8ffb0 sp=0x4004e8ffb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x4000420a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x40002190e0, 0x4, 0x1136a87, 0xf, 0x4000211e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 20 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x40001aa678) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 21 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x40001aa690) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 22 [IO wait]: internal/poll.runtime_pollWait(0xffff700cce88, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400008e098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400008e080, 0x400009f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400008e080, 0x400009f000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010010, 0x400009f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x40000820c0, 0x4000410001, 0xdff, 0xdff, 0x40001998e8, 0x6902d8, 0x400020d948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400000e048, 0x4000410001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x4000130000, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4000130000, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4000130000, 0xf5a960, 0x4000114260, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4000114240, 0x40001c27c0, 0x400020db28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400012a000, 0x12f8f98, 0x4000114240, 0x40001b5537, 0x16, 0x40003b39e0, 0x4000199c01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400012a000, 0x12f8f98, 0x4000114240, 0x400012a000, 0x40001100c0, 0x40001100d0, 0x400011c080, 0x40001180a0, 0xee0980, 0x40003b39e0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400012a000, 0x12f8f98, 0x4000114240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4000218060, 0x4000199f78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x4000143f78, 0x4000218060, 0x12db8a8, 0x400000e018, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x4000218060, 0x40001bd2c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x4000218060, 0x40001bd2c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 23 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 24 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 25 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 26 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 27 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 34 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400010c100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 29 [syscall]: syscall.Syscall(0x3f, 0x0, 0x4000408fa0, 0x4, 0x4000408e68, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x4000408fa0, 0x4, 0x4, 0x0, 0x4000116030, 0x13) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x40001ba000, 0x4000408fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x40001b8000, 0x4000408fa0, 0x4, 0x4, 0x400011c080, 0x40001180a0, 0xee0980) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 52 [IO wait]: internal/poll.runtime_pollWait(0xffff700ccda0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400008e198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400008e180, 0x400009e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400008e180, 0x400009e000, 0x1000, 0x1000, 0x4000066a88, 0x68f7a4, 0x40001e61b0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b80a8, 0x400009e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c240, 0x400009e000, 0x1000, 0x1000, 0xf12180, 0x4000466040, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001146c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001146c0, 0xa, 0x1008880, 0x4000466000, 0x199, 0xad9, 0x4000066b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40001146c0, 0x2, 0x0, 0x9, 0x56b, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400041c5d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400041c5d0, 0x400046c000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400011a040, 0x400046c000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400011a040, 0x400046c000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400011a080, 0x400046c000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40001e6160, 0x0, 0x4000066eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40001e6160, 0x0, 0x0, 0x4000066f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40001e6160, 0xee0d40, 0x4000466000, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400045c000, 0x4000114780, 0x4000182300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 50 [select]: net/http.(*persistConn).readLoop(0x400000c240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 51 [select]: net/http.(*persistConn).writeLoop(0x400000c240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 5 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000236300, 0x12c18bc, 0x1, 0x400013fe58, 0x4000182c00, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x40000b66c8, 0x4000182c00, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400013ffa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x40003f0880) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 6 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x40003f0880) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 55 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x400041a620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 9 [IO wait]: internal/poll.runtime_pollWait(0xffff700cccb8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011c118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011c100, 0x40000dc000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011c100, 0x40000dc000, 0x1000, 0x1000, 0x4000150be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000116080, 0x40000dc000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400041e6c0, 0x40000dc000, 0x1000, 0x1000, 0x4000150cf8, 0x67867c, 0x40000ce360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003c5320) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40003c5320, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400041e6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 10 [select]: net/http.(*persistConn).writeLoop(0x400041e6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 74 [select]: net/http.(*persistConn).readLoop(0x40001c65a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 76 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400011aa40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 37 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40003f0840) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 67 [IO wait]: internal/poll.runtime_pollWait(0xffff700cc1d8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000208d18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000208d00, 0x40044ca000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000208d00, 0x40044ca000, 0x1000, 0x1000, 0x40045d1688, 0x69f178, 0x40045d16f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b8588, 0x40044ca000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c65a0, 0x40044ca000, 0x1000, 0x1000, 0x5, 0x4002494000, 0x40045d17c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40045e0de0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40045e0de0, 0x68f90a, 0x400247cb00, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40045e0de0, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400142e6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400142e6c0, 0x40045f6000, 0x1000, 0x1000, 0x40045d1998, 0x44e884, 0x40045d1948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40003f0580, 0x40045f6000, 0x1000, 0x1000, 0x40024bad18, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40003f0580, 0x40045f6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40003f05c0, 0x40045f6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40045d1d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40045d1d80, 0xa, 0x40045f6201, 0x1, 0xffff99580658, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40045d1d80, 0x40045f620a, 0x1, 0x4000037710, 0x0, 0x40045f6259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40045d1d80, 0x40001e620a, 0x4000037710, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000197b10, 0x40044c54a0, 0x1f, 0x40045d1ee0, 0x40044c37d0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x4000197b10, 0x113075a, 0x7, 0x40001107c0, 0xd, 0x40044c37d0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x4000197ad0, 0x40001107c0, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 73 [select]: net/http.(*persistConn).writeLoop(0x40001c66c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 61 [select]: net/http.(*persistConn).readLoop(0x40001c6a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 58 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x400041a620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 59 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x400041a620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 60 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x400041a620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 62 [select]: net/http.(*persistConn).writeLoop(0x40001c6a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 72 [select]: net/http.(*persistConn).readLoop(0x40001c66c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 99 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x40002335e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 39 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400011a340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 13 [IO wait]: internal/poll.runtime_pollWait(0xffff700cc2c0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400008f098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400008f080, 0x40045e6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400008f080, 0x40045e6000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010240, 0x40045e6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000cb40, 0x40045e6000, 0x1000, 0x1000, 0x4001416600, 0x200000003, 0x4001416600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40045e0720) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40045e0720, 0x400047600a, 0x40045d78c8, 0x4d3b10, 0x40001ba060, 0x4000476000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40045e0720, 0x0, 0x1, 0x0, 0x40001ba078, 0x4000476000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40025e6360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40025e6360, 0x400046d000, 0x1000, 0x1000, 0x40045d79b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400011a240, 0x400046d000, 0x1000, 0x1000, 0x4000114a78, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400011a240, 0x400046d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400011a280, 0x400046d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40045d7dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40045d7dd0, 0xa, 0x400046d201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40045d7dd0, 0x400046d20a, 0x1, 0x4000110789, 0x0, 0x400046d2e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40045d7dd0, 0x40001fc20a, 0x4000110789, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000197b10, 0x400003aa50, 0x17, 0x40045d7f20, 0x4002469e00, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x4000197b10, 0x113075a, 0x7, 0x4002469e00, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x4000197ad0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 14 [IO wait]: internal/poll.runtime_pollWait(0xffff700cc660, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000208398, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000208380, 0x400259e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000208380, 0x400259e000, 0x1000, 0x1000, 0x40, 0x1d, 0x40000d97b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b8528, 0x400259e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c6a20, 0x400259e000, 0x1000, 0x1000, 0x4001416780, 0x200000003, 0x4001416780) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000115260) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000115260, 0x400047600a, 0x40000d98c8, 0x4d3b10, 0x40001ba060, 0x4000476000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000115260, 0x0, 0x1, 0x4dd090, 0x40001ba078, 0x4000476000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4002471200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4002471200, 0x400247f000, 0x1000, 0x1000, 0x40000d99b8, 0x0, 0x40000d9988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40003f1140, 0x400247f000, 0x1000, 0x1000, 0x4000001980, 0x3, 0x40000d9be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40003f1140, 0x400247f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40003f1180, 0x400247f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40000d9dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000d9dd0, 0xa, 0x400247f201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40000d9dd0, 0x400247f20a, 0x1, 0x4002460d5b, 0x0, 0x400247f263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40000d9dd0, 0x400004670a, 0x4002460d5b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000197b10, 0x40001d5bf0, 0x24, 0x40000d9f20, 0x4004508660, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x4000197b10, 0x113075a, 0x7, 0x4004508660, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x4000197ad0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 40 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x4000083740) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 41 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x4002470240) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 42 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x400008eb00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 43 [IO wait]: internal/poll.runtime_pollWait(0xffff700cc748, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400008ec18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400008ec00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400008ec00, 0x0, 0x4000407698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4002469b00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4002469b00, 0x12ed9d0, 0x4002469b00, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x400008eb80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 44 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40000a1860, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 45 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40000a1860, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 46 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x40000a1860) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 47 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x4002470d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 75 [select]: net/http.(*persistConn).writeLoop(0x40001c65a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 98 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x40002335e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 82 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x4000278770) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 97 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400011b080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 126 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x400041a9a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 125 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x400041a9a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 91 [select]: net/http.(*persistConn).readLoop(0x400000cb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 92 [select]: net/http.(*persistConn).writeLoop(0x400000cb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 77 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40003f0c40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 78 [IO wait]: internal/poll.runtime_pollWait(0xffff700cc490, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000209e18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000209e00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000209e00, 0x58ec48, 0x40025c4960, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004738978, 0x4714f4, 0x40025c4960, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004738978, 0x4000110800, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x40001d9700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 79 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x40003f1900, 0x40004652d0, 0x4000465340, 0x4004c322b0, 0x12e8f90, 0x400041a1c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x40046cfa94, 0x5, 0x40001d9700, 0x12eca70, 0x400011bf80, 0x130d478, 0x40002787e0, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x400144ee00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x400144ee00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 100 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x40003f1900, 0x4004c322b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 101 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x400041a1c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 102 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x400041a1c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 103 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x400045c5a0, 0x4000038cc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 104 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x400011aa00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 105 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x4000233680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 106 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x4000233680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 107 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x4000010420) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 108 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x4000233680, 0x4000038d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 109 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000236300, 0x1144559, 0x1b, 0x4004d1be38, 0x4000038e40, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004c32dd0, 0x4000038e40, 0xf1f700, 0x4000237fa0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004d1bf80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004cfe5a0, 0x4004c32dd0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 110 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 111 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x40001811d0, 0xd00007, 0x4004c76678, 0x5, 0x4004c32de0, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 112 [IO wait]: internal/poll.runtime_pollWait(0xffff700cc0f0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400008f218, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400008f200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400008f200, 0xffff99577a68, 0x30, 0x4004758b10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c78318, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c78318, 0x4004758b10, 0x11ae500, 0x400041a4e0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x400041a460, 0x12ed9d0, 0x4004c78318, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x400041a460, 0x12ed9d0, 0x4004c78318, 0x4000420a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 113 [IO wait]: internal/poll.runtime_pollWait(0xffff700cc008, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400008f318, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400008f300, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400008f300, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c78330, 0xffff99577a68, 0xffff70cc8498, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c78330, 0x665e1c, 0x0, 0xfe5e60, 0x40000595c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004c78348, 0x4004758ba0, 0x11ae500, 0x400041a860, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x400041a7e0, 0x12ec710, 0x4004c78348, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x400041a7e0, 0x12ec710, 0x4004c78348, 0x4000420a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 127 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x400041a9a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 123 [semacquire]: sync.runtime_Semacquire(0x4004dc1ae8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004dc1ae0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x40044fb600, 0x400010e570, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffe7726969, 0x29, 0xffffe77269cb, 0x25, 0x40047774a0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004762780, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffe7726969, 0x29, 0xffffe77269cb, 0x25, 0x40047774a0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004762780, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004e8f190, 0x4004e8ee60, 0x400010e570, 0x400242c000, 0x40003f0840, 0x0, 0x17f, 0x40045d2da8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x4000420a80, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004762780, 0xb, 0x4004762778, 0x6, 0x4004762790, 0xd, 0x40047367a0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x4000420a80, 0x40003c0000, 0x400041c7e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x4000420a80, 0x400006d460, 0x2, 0x2, 0x400006d470, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 148 [IO wait]: internal/poll.runtime_pollWait(0xffff6959fd00, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400011d218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400011d200, 0x4004791000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400011d200, 0x4004791000, 0x1000, 0x1000, 0x4004d00be0, 0x469b10, 0x4001417b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4001448190, 0x4004791000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400041efc0, 0x4004791000, 0x1000, 0x1000, 0x4004d00cf8, 0x67867c, 0x400478c6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40000830e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40000830e0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400041efc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 149 [select]: net/http.(*persistConn).writeLoop(0x400041efc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 146 [IO wait]: internal/poll.runtime_pollWait(0xffff700cc578, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000208e98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000208e80, 0x40044c8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000208e80, 0x40044c8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b8490, 0x40044c8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c66c0, 0x40044c8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40045e0d20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40045e0d20, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40045e0d20, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40047883c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40047883c0, 0x4004784800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400143b400, 0x4004784800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400143b400, 0x4004784800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400143b440, 0x4004784800, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40047ca6e0, 0x0, 0x4004d11eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40047ca6e0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40047ca6e0, 0xee0d40, 0x400145a5f0, 0x12db068, 0x400143b440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004674c60, 0x4000083020, 0x400478c4e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 121 [select]: net/http.(*persistConn).readLoop(0x4004c7a5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 122 [select]: net/http.(*persistConn).writeLoop(0x4004c7a5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 153 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4001433c30, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 152 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4001433c30, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 124 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x400041a9a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 154 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4001433c30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 162 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x400041a9a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 163 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000236300, 0x1144559, 0x1b, 0x4004d17e38, 0x4004d22f00, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x400456e0d0, 0x4004d22f00, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004d17f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004cff380, 0x400456e0d0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 128 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4004d680c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 179 [IO wait]: internal/poll.runtime_pollWait(0xffff6959fc18, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400008f818, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400008f800, 0x4004d57000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400008f800, 0x4004d57000, 0x1000, 0x1000, 0x4004d02be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010530, 0x4004d57000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004c7a7e0, 0x4004d57000, 0x1000, 0x1000, 0x4004d02cf8, 0x67867c, 0x4004d23620) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004cff680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004cff680, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004c7a7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 180 [select]: net/http.(*persistConn).writeLoop(0x4004c7a7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 164 [IO wait]: internal/poll.runtime_pollWait(0xffff6959fde8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400008f498, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400008f480, 0x4004d55000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400008f480, 0x4004d55000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000104c8, 0x4004d55000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004c7a5a0, 0x4004d55000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004cff1a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004cff1a0, 0x100a, 0x0, 0x0, 0x0, 0x0, 0x40000b6438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004cff1a0, 0x41ab8c, 0xffff69616c00, 0x100100000000, 0xffff697eafff, 0x400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400456a360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400456a360, 0x4004539200, 0x200, 0x200, 0x40000b6508, 0x41cff4, 0xffff69616c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000097340, 0x4004539200, 0x200, 0x200, 0xffff70cb0170, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000097340, 0x4004539200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000097380, 0x4004539200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004578000, 0x4001417e00, 0x40000b66b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004578000, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004578000, 0xee0d40, 0x400012a780, 0x12db068, 0x4000097380) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004562870, 0x4001441200, 0x40045740c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 132 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4004d68100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 133 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x400247dc30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 134 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x400247dc30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 135 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x400247dc30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 136 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x400247dc30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 137 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4000213100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 138 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x4000213100, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 139 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x4000420a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 140 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x4000420a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 141 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x4000420a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 142 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:51:23.150+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:51:23.151+00:00 [Info] Setting ipv6=false 2021-08-16T13:51:23.161+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:51:23.163+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:51:23.163+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:51:23.163+00:00 [Info] Setting log level to Info 2021-08-16T13:51:23.164+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:51:23.166+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:51:23 audit: created new audit service 2021-08-16T13:51:23.174+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:51:23.174+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:51:23.174+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:51:23.174+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:51:23.174+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:51:23.186+00:00 [Info] security context initialized 2021-08-16T13:51:23.186+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:51:23.186+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:51:23.187+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:51:23.187+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:51:23.187+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:51:23.201+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:51:23.201+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:51:23.203+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:51:23.228+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:51:23.230+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:51:23.230+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:51:23.231+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:51:23.232+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:51:23.232+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:51:23.233+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:51:23.236+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:51:23.279+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:51:23.279+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 53 us 2021-08-16T13:51:23.279+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:51:23.282+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:51:23.282+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:51:23.284+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:51:23.284+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:51:23.284+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:51:23.284+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:51:23.284+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:51:23.286+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:23.286+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:23.287+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:51:23.287+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:51:23.287+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:51:23.287+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:23.287+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:51:23.287+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:51:23.287+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:51:23.287+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:51:23.287+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:51:23.287+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:51:23.287+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:51:23.294+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 6.549194ms 2021-08-16T13:51:23.294+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:51:23.295+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:51:23.295+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:23.295+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:51:23.295+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:51:23.295+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:51:23.295+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... unexpected fault address 0x40013dfb1f fatal error: fault 2021-08-16T13:51:23.306+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:51:23.306+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:51:23.306+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:51:23.306+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:51:23.307+00:00 [Info] : Recover shard shard1 from metadata completed. [signal SIGBUS: bus error code=0x1 addr=0x40013dfb1f pc=0xbd8ea4] goroutine 171 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004e389f0 sp=0x4004e389c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004e38a30 sp=0x4004e389f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x400036dd40, 0x40046c0c00, 0x11a9890, 0x4004e38b40, 0x400036dd58, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004e38ab0 sp=0x4004e38a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x400036dd40, 0x40046c0c00, 0x11a9890, 0x4004e38b40, 0x400036dd58, 0xc18f8c, 0x4004e38c18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004e38af0 sp=0x4004e38ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4000374a00, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004e38da0 sp=0x4004e38af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4000374a00, 0x40013bd500, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004e38e60 sp=0x4004e38da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x40044edbd0, 0x40044edbe0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004e39200 sp=0x4004e38e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x40044edbd0, 0x40044edbe0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004e398f0 sp=0x4004e39200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004d45be0, 0x4004f0e700, 0x4002559600, 0x40044edbb0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004e39fb0 sp=0x4004e398f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004e39fb0 sp=0x4004e39fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x4000132000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x40000fbdd0, 0x4, 0x1136a87, 0xf, 0x40000f9e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 7 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400000e690) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 8 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400000e6a8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 9 [IO wait]: internal/poll.runtime_pollWait(0xffff8c5c5198, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0898, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0880, 0x40001a3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0880, 0x40001a3000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000105b0, 0x40001a3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x4000082cc0, 0x40001a8001, 0xdff, 0xdff, 0x400009f8e8, 0x6902d8, 0x40000f5948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400000e6f0, 0x40001a8001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40000ce9a0, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40000ce9a0, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40000ce9a0, 0xf5a960, 0x4000082da0, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4000082d80, 0x40000467c0, 0x40000f5b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400008c820, 0x12f8f98, 0x4000082d80, 0x4000037447, 0x16, 0x400000d560, 0x400009fc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400008c820, 0x12f8f98, 0x4000082d80, 0x400008c820, 0x4000036c58, 0x4000036c60, 0x40000f0900, 0x40000466e0, 0xee0980, 0x400000d560, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400008c820, 0x12f8f98, 0x4000082d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x40000fa3c0, 0x400009ff78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x40000a1f78, 0x40000fa3c0, 0x12db8a8, 0x400000e6c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x40000fa3c0, 0x40000b12c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x40000fa3c0, 0x40000b12c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 11 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 12 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 13 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 14 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 15 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 18 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400010c100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 16 [syscall]: syscall.Syscall(0x3f, 0x0, 0x4000053fa0, 0x4, 0x4000053e68, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x4000053fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x4000053fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x4000053fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 37 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400011ab80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 35 [select]: net/http.(*persistConn).readLoop(0x4000176120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 36 [select]: net/http.(*persistConn).writeLoop(0x4000176120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 50 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001948e0, 0x12c18bc, 0x1, 0x40000f5e58, 0x400012e6c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x40023456c8, 0x400012e6c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40000f5fa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x400011a040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 51 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x400011a040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 57 [select]: net/http.(*persistConn).writeLoop(0x4000176fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 54 [IO wait]: internal/poll.runtime_pollWait(0xffff8c5c4fc8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0080, 0x4000126000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0080, 0x4000126000, 0x1000, 0x1000, 0x40013a5be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000102f8, 0x4000126000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001b85a0, 0x4000126000, 0x1000, 0x1000, 0x40013a5cf8, 0x67867c, 0x400012ecc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000082a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000082a20, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40001b85a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 55 [select]: net/http.(*persistConn).writeLoop(0x40001b85a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 24 [IO wait]: internal/poll.runtime_pollWait(0xffff8c5c50b0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000178118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000178100, 0x40001a9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000178100, 0x40001a9000, 0x1000, 0x1000, 0x40003b8a88, 0x68f7a4, 0x4000322050) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000107d0, 0x40001a9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000176120, 0x40001a9000, 0x1000, 0x1000, 0xf12180, 0x40001122c0, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400033d560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400033d560, 0xa, 0x1008880, 0x4000112280, 0x199, 0x1000, 0x40003b8b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400033d560, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40001307e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40001307e0, 0x400034e000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400031c000, 0x400034e000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400031c000, 0x400034e000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400031c040, 0x400034e000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4000322000, 0x0, 0x40003b8eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4000322000, 0x0, 0x0, 0x40003b8f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4000322000, 0xee0d40, 0x4000112280, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400017c000, 0x400010e900, 0x4000320000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 58 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400031cd80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 56 [select]: net/http.(*persistConn).readLoop(0x4000176fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 31 [IO wait]: internal/poll.runtime_pollWait(0xffff8c5c4888, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40013d4098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40013d4080, 0x40013c5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40013d4080, 0x40013c5000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000114040, 0x40013c5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c480, 0x40013c5000, 0x1000, 0x1000, 0x4000377200, 0x200000003, 0x4000377200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40013b45a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40013b45a0, 0x400038400a, 0x40000f38c8, 0x4d3b10, 0x40000820c0, 0x4000384000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40013b45a0, 0x0, 0x1, 0x0, 0x40000820d8, 0x4000384000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4002414780) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4002414780, 0x40013f4000, 0x1000, 0x1000, 0x40000f39b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400011a500, 0x40013f4000, 0x1000, 0x1000, 0x40013b4c58, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400011a500, 0x40013f4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400011a540, 0x40013f4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40000f3dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000f3dd0, 0xa, 0x40013f4201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40000f3dd0, 0x40013f420a, 0x1, 0x40013abe59, 0x0, 0x40013f42e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40000f3dd0, 0x400009920a, 0x40013abe59, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000098fb0, 0x400016a3a8, 0x17, 0x40000f3f20, 0x400039f7d0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x4000098fb0, 0x113075a, 0x7, 0x400039f7d0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x4000098f70) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 123 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x400243f5e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 65 [select]: net/http.(*persistConn).writeLoop(0x40003da6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 32 [IO wait]: internal/poll.runtime_pollWait(0xffff8c5c4a58, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000178c98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000178c80, 0x4000128000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000178c80, 0x4000128000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010320, 0x4000128000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000176fc0, 0x4000128000, 0x1000, 0x1000, 0x4000103800, 0x200000003, 0x4000103800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000082b40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000082b40, 0x400038400a, 0x40000f98c8, 0x4d3b10, 0x40000820c0, 0x4000384000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000082b40, 0x0, 0x1, 0x0, 0x40000820d8, 0x4000384000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40013acba0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40013acba0, 0x40003a6000, 0x1000, 0x1000, 0x40000f99b8, 0x0, 0x40000f9988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000081bc0, 0x40003a6000, 0x1000, 0x1000, 0x4000103200, 0x3, 0x40000f9be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000081bc0, 0x40003a6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000081c00, 0x40003a6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40000f9dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000f9dd0, 0xa, 0x40003a6201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40000f9dd0, 0x40003a620a, 0x1, 0x40013abc8b, 0x0, 0x40003a6263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40000f9dd0, 0x40013be20a, 0x40013abc8b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000098fb0, 0x4000166540, 0x24, 0x40000f9f20, 0x400039f7b8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x4000098fb0, 0x113075a, 0x7, 0x400039f7b8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x4000098f70) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 60 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x400243e380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 80 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40044a0b40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 66 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x400243e380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 67 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x400243e380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 68 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x400243e380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 70 [select]: net/http.(*persistConn).readLoop(0x400000c480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 71 [select]: net/http.(*persistConn).writeLoop(0x400000c480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 122 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x400243f5e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 38 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x400033dbc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 39 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x400036b7a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 40 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x40000f1100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 41 [IO wait]: internal/poll.runtime_pollWait(0xffff8c5c47a0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1218, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f1200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f1200, 0x0, 0x40013d0e98, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4002509b78, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4002509b78, 0x12ed9d0, 0x4002509b78, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x40000f1180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 42 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4000138d00, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 43 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4000138d00, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 44 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x4000138d00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 45 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x40001305a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 99 [select]: net/http.(*persistConn).writeLoop(0x400000cd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 64 [select]: net/http.(*persistConn).readLoop(0x40003da6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 48 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40001f47e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 98 [IO wait]: internal/poll.runtime_pollWait(0xffff8c5c45d0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40003a4298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40003a4280, 0x40023fa000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40003a4280, 0x40023fa000, 0x1000, 0x1000, 0x400006abe0, 0x469b10, 0x4000377500) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000328098, 0x40023fa000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000cd80, 0x40023fa000, 0x1000, 0x1000, 0x400006acf8, 0x67867c, 0x4002520960) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40023dec60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40023dec60, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000cd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 85 [IO wait]: internal/poll.runtime_pollWait(0xffff8c5c46b8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1398, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f1380, 0x40023f8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f1380, 0x40023f8000, 0x1000, 0x1000, 0x40024a3688, 0x69f178, 0x40024a36f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000328088, 0x40023f8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003da6c0, 0x40023f8000, 0x1000, 0x1000, 0x5, 0x4000148500, 0x40024a37c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40023de5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40023de5a0, 0x68f90a, 0x40003c2c60, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40023de5a0, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4002414990) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4002414990, 0x40013f5000, 0x1000, 0x1000, 0x40024a3998, 0x44e884, 0x40024a3948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400011a640, 0x40013f5000, 0x1000, 0x1000, 0x400010f498, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400011a640, 0x40013f5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400011a680, 0x40013f5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40024a3d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40024a3d80, 0xa, 0x40013f5201, 0x1, 0xffff8c5dfa08, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40024a3d80, 0x40013f520a, 0x1, 0x40046267b0, 0x0, 0x40013f5259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40024a3d80, 0x4000323a0a, 0x40046267b0, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000098fb0, 0x40003a06c0, 0x1f, 0x40024a3ee0, 0x400039fd88, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x4000098fb0, 0x113075a, 0x7, 0x40013abe70, 0xd, 0x400039fd88, 0x0, 0x0, 0x2) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x4000098f70, 0x40013abe70, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 115 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400011a7c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 49 [select]: net/http.(*persistConn).readLoop(0x400000d0e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 114 [select]: net/http.(*persistConn).writeLoop(0x400000d0e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 88 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001f77c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 93 [IO wait]: internal/poll.runtime_pollWait(0xffff8c5c4970, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40003a4b98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40003a4b80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40003a4b80, 0x12d9d48, 0x40044a0c00, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40046ec210, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40046ec210, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x40044a0e40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 121 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x400243f5e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 113 [IO wait]: internal/poll.runtime_pollWait(0xffff8c36afd8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004766e18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004766e00, 0x4004cc8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004766e00, 0x4004cc8000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40003d23c8, 0x4004cc8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003dbb00, 0x4004cc8000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004cbae40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004cbae40, 0x100a, 0x40013edbe8, 0x48731c, 0x9, 0x1, 0x40013edc38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004cbae40, 0x41ab8c, 0xffff8c1b0e00, 0x100100000000, 0xffff8c33cfff, 0x400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004e34d50) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004e34d50, 0x400250fa00, 0x200, 0x200, 0x40013edd08, 0x41cff4, 0xffff8c1b0e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400031cd40, 0x400250fa00, 0x200, 0x200, 0x400250fa00, 0x200, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400031cd40, 0x400250fa00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400031cdc0, 0x400250fa00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004e7c000, 0x680d00, 0x40013edeb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004e7c000, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004e7c000, 0xee0d40, 0x4000112cd0, 0x12db068, 0x400031cdc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40023dd7a0, 0x4004e74120, 0x4004e7a060) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 94 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x400011ba40, 0x40013bcd90, 0x40013bce00, 0x40045c6b60, 0x12e8f90, 0x400255c700, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x40045eae54, 0x5, 0x40044a0e40, 0x12eca70, 0x400011b880, 0x130d478, 0x4000174070, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40013e67e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40013e67e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 81 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x400012ad20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 130 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x400012ad20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 131 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x400011ba40, 0x40045c6b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 132 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x400255c700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 133 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x400255c700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 134 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x4002416990, 0x40045bb7a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 135 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x400011a780) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 136 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x400012adc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 137 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x400012adc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 138 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x4000010aa8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 139 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x400012adc0, 0x40045bb800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 140 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001948e0, 0x1144559, 0x1b, 0x400246de38, 0x40045bb920, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x40045c70c0, 0x40045bb920, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400246df80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4002419e00, 0x40045c70c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 141 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 142 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400008cff0, 0xd00007, 0x40045ebc18, 0x5, 0x40045c70d0, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 143 [IO wait]: internal/poll.runtime_pollWait(0xffff8c5c4400, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40013d5198, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40013d5180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40013d5180, 0xffffb44a2f18, 0x30, 0x400238c870) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40045f3050, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40045f3050, 0x400238c870, 0x11ae500, 0x40013e6940, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40013e68c0, 0x12ed9d0, 0x40045f3050, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40013e68c0, 0x12ed9d0, 0x40045f3050, 0x4000132000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 144 [IO wait]: internal/poll.runtime_pollWait(0xffff8c5c4318, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40013d5298, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40013d5280, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40013d5280, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40045f3068, 0xffffb44a2a68, 0xffff8c0d4208, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40045f3068, 0x665e1c, 0x0, 0xfe5e60, 0x40013ee5c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x40045f3080, 0x4004c7e1e0, 0x11ae500, 0x40013e6a20, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40013e69a0, 0x12ec710, 0x40045f3080, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40013e69a0, 0x12ec710, 0x40045f3080, 0x4000132000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 120 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x400243f5e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 106 [IO wait]: internal/poll.runtime_pollWait(0xffff8c36b0c0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000178798, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000178780, 0x4004e3e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000178780, 0x4004e3e000, 0x1000, 0x1000, 0x40024bebe0, 0x469b10, 0x4000377680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000328188, 0x4004e3e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40023fe000, 0x4004e3e000, 0x1000, 0x1000, 0x40024becf8, 0x67867c, 0x4002521140) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40023df680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40023df680, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40023fe000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 107 [select]: net/http.(*persistConn).writeLoop(0x40023fe000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 117 [IO wait]: internal/poll.runtime_pollWait(0xffff8c5c44e8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40003a4418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40003a4400, 0x4000173000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40003a4400, 0x4000173000, 0x1000, 0x1000, 0x1, 0x4004c80540, 0x400006ba78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40003d20d0, 0x4000173000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000d0e0, 0x4000173000, 0x1000, 0x1000, 0xffff8c1a2d00, 0x100100000000, 0xffff8c33cfff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400010efc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400010efc0, 0x40a, 0x400, 0x200, 0xffff8c1a2d00, 0xffff8c33cfff, 0x32) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400010efc0, 0x1, 0x40013ec458, 0x41c330, 0xffff8c0813b0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004c80900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004c80900, 0x4004cb8000, 0x200, 0x200, 0x40013ec508, 0x412ddc, 0xffff8c0813b0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004768c40, 0x4004cb8000, 0x200, 0x200, 0xffffb44a2108, 0x35, 0x4004cb8000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004768c40, 0x4004cb8000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004768c80, 0x4004cb8000, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004bfd760, 0x40045bb900, 0x40013ec6b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004bfd760, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004bfd760, 0xee0d40, 0x40003e2e60, 0x12db068, 0x4004768c80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004cb6090, 0x4004629e60, 0x4004763920) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 119 [semacquire]: sync.runtime_Semacquire(0x4004d45be8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004d45be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x4002559600, 0x4000097260, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffc3a8a969, 0x29, 0xffffc3a8a9cb, 0x25, 0x40046e5260, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x40046c9a40, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffc3a8a969, 0x29, 0xffffc3a8a9cb, 0x25, 0x40046e5260, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x40046c9a40, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004e39190, 0x4004e38e60, 0x4000097260, 0x4002388900, 0x400031cd80, 0x0, 0x17f, 0x400009cda8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x4000132000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x40046c9a40, 0xb, 0x40046c9a38, 0x6, 0x40046c9a50, 0xd, 0x40046d3400, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x4000132000, 0x4000330000, 0x4000130030) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x4000132000, 0x40003b0d70, 0x2, 0x2, 0x40003b0d80, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 96 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4004768f80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 124 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4004768f40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 152 [IO wait]: internal/poll.runtime_pollWait(0xffff8c36b1a8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004766d18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004766d00, 0x4004cca000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004766d00, 0x4004cca000, 0x1000, 0x1000, 0x4004cd1be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40003d23d0, 0x4004cca000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003db9e0, 0x4004cca000, 0x1000, 0x1000, 0x4004cd1cf8, 0x67867c, 0x4004cbeae0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004cbaf60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004cbaf60, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40003db9e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 127 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004765290, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 128 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004765290, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 129 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4004765290) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 146 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x400243f5e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 147 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001948e0, 0x1144559, 0x1b, 0x4004e2de38, 0x4004763da0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004c82860, 0x4004763da0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004e2df80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004cba180, 0x4004c82860) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 153 [select]: net/http.(*persistConn).writeLoop(0x40003db9e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 150 [select]: net/http.(*persistConn).readLoop(0x40003dbb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 151 [select]: net/http.(*persistConn).writeLoop(0x40003dbb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 97 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x4004765130) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 162 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x4004765130) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 163 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x4004765130) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 164 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x4004765130) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 165 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x40046d5100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 166 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x40046d5100, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 167 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x4000132000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 168 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x4000132000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 169 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x4000132000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 170 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:51:28.475+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:51:28.476+00:00 [Info] Setting ipv6=false 2021-08-16T13:51:28.480+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:51:28.483+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:51:28.483+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:51:28.483+00:00 [Info] Setting log level to Info 2021-08-16T13:51:28.483+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:51:28.486+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:51:28 audit: created new audit service 2021-08-16T13:51:28.495+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:51:28.495+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:51:28.495+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:51:28.495+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:51:28.495+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:51:28.505+00:00 [Info] security context initialized 2021-08-16T13:51:28.505+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:51:28.505+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:51:28.505+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:51:28.505+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:51:28.505+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:51:28.518+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:51:28.523+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:51:28.529+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:51:28.553+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:51:28.553+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:51:28.554+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:51:28.555+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:51:28.556+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:51:28.558+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:51:28.558+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:51:28.559+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:51:28.609+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:51:28.609+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 62 us 2021-08-16T13:51:28.609+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:51:28.611+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:51:28.612+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:51:28.615+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:51:28.615+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:51:28.615+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:51:28.615+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:51:28.615+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:51:28.615+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:28.616+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:28.618+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:51:28.618+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:51:28.618+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:51:28.618+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:28.618+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:51:28.618+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:51:28.618+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:51:28.618+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:51:28.618+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:51:28.618+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:51:28.618+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:51:28.623+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 5.291637ms 2021-08-16T13:51:28.624+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:51:28.625+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:51:28.625+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:28.625+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:51:28.625+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:51:28.625+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:51:28.625+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:51:28.635+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:51:28.635+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:51:28.635+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:51:28.635+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards unexpected fault address 0x4004765d5f fatal error: fault 2021-08-16T13:51:28.635+00:00 [Info] : Recover shard shard1 from metadata completed. [signal SIGBUS: bus error code=0x1 addr=0x4004765d5f pc=0xbd8ea4] goroutine 173 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004eac9f0 sp=0x4004eac9c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004eaca30 sp=0x4004eac9f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x4004c7a820, 0x4004e08c00, 0x11a9890, 0x4004eacb40, 0x4004c7a838, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004eacab0 sp=0x4004eaca40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x4004c7a820, 0x4004e08c00, 0x11a9890, 0x4004eacb40, 0x4004c7a838, 0xc18f8c, 0x4004eacc18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004eacaf0 sp=0x4004eacab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4000032f00, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004eacda0 sp=0x4004eacaf0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4000032f00, 0x40023e5810, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004eace60 sp=0x4004eacda0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004c79450, 0x4004c79460, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004ead200 sp=0x4004eace60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004c79450, 0x4004c79460, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004ead8f0 sp=0x4004ead200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004e41fc0, 0x4004e8ae00, 0x4000169b80, 0x4004c79430) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004eadfb0 sp=0x4004ead8f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004eadfb0 sp=0x4004eadfb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x400143a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x40001198f0, 0x4, 0x1136a87, 0xf, 0x4000125e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 18 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x4000126120) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 19 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x4000126138) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 20 [IO wait]: internal/poll.runtime_pollWait(0xffff707b7758, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0100, 0x40000f9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0100, 0x40000f9000, 0x1000, 0x1000, 0x1311860, 0x47b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010500, 0x40000f9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x4000082420, 0x40001a8001, 0xdff, 0xdff, 0x40000a18e8, 0x6902d8, 0x4000121948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400000e5a0, 0x40001a8001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40001a0000, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40001a0000, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40001a0000, 0xf5a960, 0x4000196080, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4000196060, 0x40001a20e0, 0x4000121b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x4000194000, 0x12f8f98, 0x4000196060, 0x4000186867, 0x16, 0x40001ac120, 0x40000a1c01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x4000194000, 0x12f8f98, 0x4000196060, 0x4000194000, 0x4000186078, 0x4000186080, 0x400019e000, 0x40001a2000, 0xee0980, 0x40001ac120, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x4000194000, 0x12f8f98, 0x4000196060) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4000118870, 0x40000a1f78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400009df78, 0x4000118870, 0x12db8a8, 0x400000e570, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x4000118870, 0x4000142280, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x4000118870, 0x4000142280) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 21 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 22 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 23 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 24 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 25 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 7 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400000a800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 35 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4001444b40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 26 [syscall]: syscall.Syscall(0x3f, 0x0, 0x4000055fa0, 0x4, 0x4000055e68, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x4000055fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x4000055fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x4000055fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 11 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400011c7d0, 0x12c18bc, 0x1, 0x400009fe58, 0x40014366c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x40001ceec8, 0x40014366c0, 0x40001ceec8, 0x679704) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400009ffa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4001444000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 9 [select]: net/http.(*persistConn).readLoop(0x40023cc120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 10 [select]: net/http.(*persistConn).writeLoop(0x40023cc120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 12 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4001444000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 66 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000188180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 15 [IO wait]: internal/poll.runtime_pollWait(0xffff707b7588, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40023d0298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40023d0280, 0x40000fe000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40023d0280, 0x40000fe000, 0x1000, 0x1000, 0x4000066be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400143e068, 0x40000fe000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40023cc5a0, 0x40000fe000, 0x1000, 0x1000, 0x4000066cf8, 0x67867c, 0x4001436e40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4001434840) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4001434840, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40023cc5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 16 [select]: net/http.(*persistConn).writeLoop(0x40023cc5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 54 [IO wait]: internal/poll.runtime_pollWait(0xffff707b7670, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40023d0118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40023d0100, 0x40000f8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40023d0100, 0x40000f8000, 0x1000, 0x1000, 0x40023e2a88, 0x68f7a4, 0x400013e730) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400143e020, 0x40000f8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40023cc120, 0x40000f8000, 0x1000, 0x1000, 0xf12180, 0x400013a1d0, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4001434480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4001434480, 0xa, 0x1008880, 0x400013a190, 0x199, 0xad9, 0x40023e2b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4001434480, 0x2, 0x0, 0x9, 0x56b, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40003c4960) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40003c4960, 0x40023ea000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000130040, 0x40023ea000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000130040, 0x40023ea000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000130080, 0x40023ea000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x400013e6e0, 0x0, 0x40023e2eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400013e6e0, 0x0, 0x0, 0x40023e2f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400013e6e0, 0xee0d40, 0x400013a190, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4000157200, 0x4000279080, 0x4000154240) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 57 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x40003ce380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 55 [select]: net/http.(*persistConn).readLoop(0x40001ac900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 56 [select]: net/http.(*persistConn).writeLoop(0x40001ac900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 61 [select]: net/http.(*persistConn).readLoop(0x400000cb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 91 [select]: net/http.(*persistConn).readLoop(0x40023cc7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 93 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000131380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 88 [IO wait]: internal/poll.runtime_pollWait(0xffff707b6aa8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0618, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0600, 0x4001458000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0600, 0x4001458000, 0x1000, 0x1000, 0x40001e1688, 0x69f178, 0x40001e16f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000105e0, 0x4001458000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40023cc7e0, 0x4001458000, 0x1000, 0x1000, 0x5, 0x4002540a00, 0x40001e17c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000197440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000197440, 0x68f90a, 0x40023d2a50, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000197440, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40001e74a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40001e74a0, 0x40024ab000, 0x1000, 0x1000, 0x40001e1998, 0x44e884, 0x40001e1948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40001f4340, 0x40024ab000, 0x1000, 0x1000, 0x4000082f58, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40001f4340, 0x40024ab000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001f4380, 0x40024ab000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40001e1d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001e1d80, 0xa, 0x40024ab201, 0x1, 0xffff9865e988, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40001e1d80, 0x40024ab20a, 0x1, 0x4004599740, 0x0, 0x40024ab259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40001e1d80, 0x40001a160a, 0x4004599740, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000098520, 0x4001446540, 0x1f, 0x40001e1ee0, 0x4001432b40, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x4000098520, 0x113075a, 0x7, 0x40024558c0, 0xd, 0x4001432b40, 0x0, 0x0, 0x2) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x40000984e0, 0x40024558c0, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 68 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x40003ce380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 107 [select]: net/http.(*persistConn).writeLoop(0x4000206c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 58 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x40003ce380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 59 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x40003ce380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 62 [select]: net/http.(*persistConn).writeLoop(0x400000cb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 36 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x40001167e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 37 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x40001e62a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 38 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x400019e080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 39 [IO wait]: internal/poll.runtime_pollWait(0xffff707b6e48, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400019e198, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400019e180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400019e180, 0x0, 0x40001d1698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40001fa288, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40001fa288, 0x12ed9d0, 0x40001fa288, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x400019e100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 40 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4000098340, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 41 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4000098340, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 42 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x4000098340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 43 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x40001e6d80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 122 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x4004c400a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 123 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x4004c400a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 46 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40023ca150) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 47 [IO wait]: internal/poll.runtime_pollWait(0xffff707b7018, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0a18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0a00, 0x400018c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0a00, 0x400018c000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000128178, 0x400018c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001ac900, 0x400018c000, 0x1000, 0x1000, 0x40001c9800, 0x200000003, 0x40001c9800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000279440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000279440, 0x40023f200a, 0x40023db8c8, 0x4d3b10, 0x40000820c0, 0x40023f2000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000279440, 0x0, 0x1, 0x0, 0x40000820d8, 0x40023f2000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400250d380) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400250d380, 0x4000442000, 0x1000, 0x1000, 0x40023db9b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4001444940, 0x4000442000, 0x1000, 0x1000, 0x4000082f58, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4001444940, 0x4000442000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4001444980, 0x4000442000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40023dbdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40023dbdd0, 0xa, 0x4000442201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40023dbdd0, 0x400044220a, 0x1, 0x40024558a9, 0x0, 0x40004422e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40023dbdd0, 0x400144370a, 0x40024558a9, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000098520, 0x4000134660, 0x17, 0x40023dbf20, 0x400000fb18, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x4000098520, 0x113075a, 0x7, 0x400000fb18, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x40000984e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 48 [IO wait]: internal/poll.runtime_pollWait(0xffff707b6f30, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000132218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000132200, 0x400018e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000132200, 0x400018e000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000128180, 0x400018e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000cb40, 0x400018e000, 0x1000, 0x1000, 0x40001c9980, 0x200000003, 0x40001c9980) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000279800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000279800, 0x40023f200a, 0x40001e38c8, 0x4d3b10, 0x40000820c0, 0x40023f2000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000279800, 0x0, 0x1, 0x0, 0x40000820d8, 0x40023f2000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400250d170) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400250d170, 0x4000435000, 0x1000, 0x1000, 0x40001e39b8, 0x0, 0x40001e3988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40014447c0, 0x4000435000, 0x1000, 0x1000, 0x4001417080, 0x3, 0x40001e3be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40014447c0, 0x4000435000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4001444800, 0x4000435000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40001e3dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001e3dd0, 0xa, 0x4000435201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40001e3dd0, 0x400043520a, 0x1, 0x400012bf6b, 0x0, 0x4000435263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40001e3dd0, 0x400012c30a, 0x400012bf6b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000098520, 0x4000182660, 0x24, 0x40001e3f20, 0x40001fa630, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x4000098520, 0x113075a, 0x7, 0x40001fa630, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x40000984e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 71 [IO wait]: internal/poll.runtime_pollWait(0xffff707b6d60, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000133818, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000133800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000133800, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004b9a6a8, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004b9a6a8, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x4001445e80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 70 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000189880) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 121 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4001444a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 115 [select]: net/http.(*persistConn).readLoop(0x400000cfc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 116 [select]: net/http.(*persistConn).writeLoop(0x400000cfc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 92 [select]: net/http.(*persistConn).writeLoop(0x40023cc7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 119 [IO wait]: internal/poll.runtime_pollWait(0xffff707b6c78, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40023d0f98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40023d0f80, 0x4004440000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40023d0f80, 0x4004440000, 0x1000, 0x1000, 0x400440abe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001281e0, 0x4004440000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000d200, 0x4004440000, 0x1000, 0x1000, 0x400440acf8, 0x67867c, 0x4000155c20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40023ec600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40023ec600, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000d200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 120 [select]: net/http.(*persistConn).writeLoop(0x400000d200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 72 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x4000081d40, 0x400015b2d0, 0x400015b340, 0x400446b970, 0x12e8f90, 0x40000de460, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x400465f114, 0x5, 0x4001445e80, 0x12eca70, 0x4001445d40, 0x130d478, 0x4000264850, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x400255cee0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x400255cee0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 124 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x4000081d40, 0x400446b970) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 125 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40000de460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 126 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40000de460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 127 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x400250b5f0, 0x4004719b00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 128 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x4001444a40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 129 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x4004c40140) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 130 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x4004c40140) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 131 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x4000128510) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 132 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x4004c40140, 0x4004719b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 133 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400011c7d0, 0x1144559, 0x1b, 0x40023d9e38, 0x4004719c80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004c78490, 0x4004719c80, 0xf1f700, 0x40003fe050) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40023d9f80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004c3ea80, 0x4004c78490) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 134 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 135 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400013b270, 0xd00007, 0x4004c2f1f8, 0x5, 0x4004c784a0, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 136 [IO wait]: internal/poll.runtime_pollWait(0xffff707b69c0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004b6e598, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4004b6e580, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4004b6e580, 0xffff986aaf18, 0x30, 0x4004c7c4e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c258a8, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c258a8, 0x4004c7c4e0, 0x11ae500, 0x40003cf040, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40003cefc0, 0x12ed9d0, 0x4004c258a8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40003cefc0, 0x12ed9d0, 0x4004c258a8, 0x400143a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 137 [IO wait]: internal/poll.runtime_pollWait(0xffff707b68d8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004b6e698, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4004b6e680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4004b6e680, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c258c0, 0xffff986aaf18, 0xffff7031f3e8, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c258c0, 0x665e1c, 0x0, 0xfe5e60, 0x4004bbe5c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004c258d8, 0x4004c7c570, 0x11ae500, 0x40003cf120, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40003cf0a0, 0x12ec710, 0x4004c258d8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40003cf0a0, 0x12ec710, 0x4004c258d8, 0x400143a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 146 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x400255cfc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 100 [IO wait]: internal/poll.runtime_pollWait(0xffff702df380, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400019e398, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400019e380, 0x4004d88000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400019e380, 0x4004d88000, 0x1000, 0x1000, 0x40023ddbe0, 0x469b10, 0x400443fc80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400019a1a0, 0x4004d88000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004c2cfc0, 0x4004d88000, 0x1000, 0x1000, 0x40023ddcf8, 0x67867c, 0x40001ed2c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000083b60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000083b60, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004c2cfc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 101 [select]: net/http.(*persistConn).writeLoop(0x4004c2cfc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 106 [select]: net/http.(*persistConn).readLoop(0x4000206c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 81 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x400255cfc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 76 [IO wait]: internal/poll.runtime_pollWait(0xffff707b6b90, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40023d0e18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40023d0e00, 0x40023ee000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40023d0e00, 0x40023ee000, 0x1000, 0x1000, 0x4004454ad8, 0x412bc0, 0x4004454a38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001281b8, 0x40023ee000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000cfc0, 0x40023ee000, 0x1000, 0x1000, 0xffff9863f008, 0x0, 0x14) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40023ec180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40023ec180, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40023ec180, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004607d70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004607d70, 0x4004649400, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004bd81c0, 0x4004649400, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004bd81c0, 0x4004649400, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004bd8200, 0x4004649400, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004b9db80, 0x0, 0x40014246b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004b9db80, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004b9db80, 0xee0d40, 0x4001474aa0, 0x12db068, 0x4004bd8200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004ba4870, 0x400461ec00, 0x400457fa40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 80 [semacquire]: sync.runtime_Semacquire(0x4004e41fc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004e41fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x4000169b80, 0x40014389f0, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xfffffa971969, 0x29, 0xfffffa9719cb, 0x25, 0x4004cef1a0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x400465fe00, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xfffffa971969, 0x29, 0xfffffa9719cb, 0x25, 0x4004cef1a0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x400465fe00, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004ead190, 0x4004eace60, 0x40014389f0, 0x40001af680, 0x4000188180, 0x0, 0x1bf, 0x4004ce8da8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x400143a000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x400465fe00, 0xb, 0x400465fdf8, 0x6, 0x400465fe10, 0xd, 0x4004b97bc0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x400143a000, 0x4000270000, 0x4001438030) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x400143a000, 0x400011c4a0, 0x2, 0x2, 0x400011c4b0, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 103 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400011c7d0, 0x1144559, 0x1b, 0x40023d5e38, 0x4004bea000, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004d847e0, 0x4004bea000, 0x4004c2cfc0, 0x4000052640) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x12c2e00, 0x40001e7401) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40023d5f80, 0x12c2e00, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x400461ede0, 0x4004d847e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 102 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x400255cfc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 147 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x400255cfc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 148 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x400255cfc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 149 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4004bd8480) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 150 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004bde4d0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 151 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004bde4d0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 152 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4004bde4d0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 145 [IO wait]: internal/poll.runtime_pollWait(0xffff702df468, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400019e918, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400019e900, 0x4004d95000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400019e900, 0x4004d95000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400019a218, 0x4004d95000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000206c60, 0x4004d95000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004d96360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004d96360, 0xa, 0x4001429be8, 0x48731c, 0x9, 0x1, 0xfc) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004d96360, 0x0, 0x4001429c48, 0x57d438, 0x11320c7, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004c7d350) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004c7d350, 0x4004c2ba00, 0x200, 0x200, 0x4004c2f278, 0x4004c2f278, 0x4000042bd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004c765c0, 0x4004c2ba00, 0x200, 0x200, 0xb2fb2d0d67, 0x24b50832, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004c765c0, 0x4004c2ba00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004c76600, 0x4004c2ba00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004ceb600, 0x0, 0x4001429eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004ceb600, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004ceb600, 0xee0d40, 0x400013b9f0, 0x12db068, 0x4004c76600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004c37440, 0x4004c3fb60, 0x4004cee900) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 162 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4004bd84c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 163 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x4004bde370) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 164 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x4004bde370) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 165 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x4004bde370) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 166 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x4004bde370) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 167 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004e36a00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 168 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x4004e36a00, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 169 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x400143a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 170 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x400143a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 171 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x400143a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 172 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:51:33.789+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:51:33.789+00:00 [Info] Setting ipv6=false 2021-08-16T13:51:33.795+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:51:33.798+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:51:33.798+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:51:33.798+00:00 [Info] Setting log level to Info 2021-08-16T13:51:33.798+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:51:33.801+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:51:33 audit: created new audit service 2021-08-16T13:51:33.809+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:51:33.809+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:51:33.810+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:51:33.810+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:51:33.810+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:51:33.820+00:00 [Info] security context initialized 2021-08-16T13:51:33.820+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:51:33.820+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:51:33.820+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:51:33.820+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:51:33.820+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:51:33.835+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:51:33.835+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:51:33.836+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:51:33.881+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:51:33.881+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:51:33.882+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:51:33.884+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:51:33.886+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:51:33.886+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:51:33.887+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:51:33.889+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:51:33.953+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:51:33.954+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 58 us 2021-08-16T13:51:33.954+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:51:33.956+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:51:33.957+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:51:33.957+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:51:33.958+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:51:33.958+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:51:33.958+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:51:33.958+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:51:33.960+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:33.960+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:33.961+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:51:33.961+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:51:33.961+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:51:33.962+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:33.962+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:51:33.962+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:51:33.962+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:51:33.962+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:51:33.962+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:51:33.962+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:51:33.962+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:51:33.969+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 6.9412ms 2021-08-16T13:51:33.969+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:51:33.971+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:51:33.971+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:33.971+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:51:33.971+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:51:33.971+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:51:33.971+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:51:33.983+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:51:33.983+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:51:33.983+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:51:33.983+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards unexpected fault address 0x4004cfe01f fatal error: fault 2021-08-16T13:51:33.984+00:00 [Info] RebalanceMgr::initService Init 2021-08-16T13:51:33.984+00:00 [Info] : Recover shard shard1 from metadata completed. [signal SIGBUS: bus error code=0x1 addr=0x4004cfe01f pc=0xbd8ea4] goroutine 190 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004c909f0 sp=0x4004c909c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004c90a30 sp=0x4004c909f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x40044d1860, 0x40044fd200, 0x11a9890, 0x4004c90b40, 0x40044d1878, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004c90ab0 sp=0x4004c90a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x40044d1860, 0x40044fd200, 0x11a9890, 0x4004c90b40, 0x40044d1878, 0xffff97cf1db8, 0xffff4fcb1d30, 0x49) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004c90af0 sp=0x4004c90ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x40000bc500, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004c90da0 sp=0x4004c90af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x40000bc500, 0x40000976c0, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004c90e60 sp=0x4004c90da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x400244df80, 0x400244df90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004c91200 sp=0x4004c90e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x400244df80, 0x400244df90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004c918f0 sp=0x4004c91200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004cd10d0, 0x4004cad180, 0x40000df080, 0x400244df60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004c91fb0 sp=0x4004c918f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004c91fb0 sp=0x4004c91fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x4000214000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x400009f680, 0x4, 0x1136a87, 0xf, 0x4000199e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 20 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x40000ac0f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 21 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x40000ac108) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 22 [IO wait]: internal/poll.runtime_pollWait(0xffff97c30758, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000190198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000190180, 0x4000387000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000190180, 0x4000387000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000a8090, 0x4000387000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x400009a780, 0x40001a0001, 0xdff, 0xdff, 0x40001178e8, 0x6902d8, 0x4000195948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x40000ac138, 0x40001a0001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x400016e6e0, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400016e6e0, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400016e6e0, 0xf5a960, 0x4000138620, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4000138600, 0x4000148340, 0x4000195b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400014a140, 0x12f8f98, 0x4000138600, 0x4000132c87, 0x16, 0x400014f0e0, 0x4000117c01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400014a140, 0x12f8f98, 0x4000138600, 0x400014a140, 0x4000132498, 0x40001324a0, 0x4000190200, 0x4000148260, 0xee0980, 0x400014f0e0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400014a140, 0x12f8f98, 0x4000138600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x400009e4e0, 0x4000117f78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400048bf78, 0x400009e4e0, 0x12db8a8, 0x4000128588, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x400009e4e0, 0x40000ba460, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x400009e4e0, 0x40000ba460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 23 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 24 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 25 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 26 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 27 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 37 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400010e200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 28 [syscall]: syscall.Syscall(0x3f, 0x0, 0x400013afa0, 0x4, 0x400013ae68, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x400013afa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000138000, 0x400013afa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000136000, 0x400013afa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 67 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000202980) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 8 [select]: net/http.(*persistConn).readLoop(0x400000c240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 9 [select]: net/http.(*persistConn).writeLoop(0x400000c240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 38 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40000a2760, 0x12c18bc, 0x1, 0x4000487e58, 0x4000038720, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x40001cdec8, 0x4000038720, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000487fa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4000202040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 39 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4000202040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 114 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40004fcb60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 43 [IO wait]: internal/poll.runtime_pollWait(0xffff97c30588, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000aa098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000aa080, 0x4000255000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000aa080, 0x4000255000, 0x1000, 0x1000, 0x4000064be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000a8058, 0x4000255000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400014e6c0, 0x4000255000, 0x1000, 0x1000, 0x4000064cf8, 0x67867c, 0x4000100480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000138f00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000138f00, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400014e6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 42 [IO wait]: internal/poll.runtime_pollWait(0xffff97c30670, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000250118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000250100, 0x4000254000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000250100, 0x4000254000, 0x1000, 0x1000, 0x40001b9a88, 0x68f7a4, 0x400016e310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010040, 0x4000254000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c240, 0x4000254000, 0x1000, 0x1000, 0xf12180, 0x400009c2c0, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400007e480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400007e480, 0xa, 0x1008880, 0x400009c280, 0x199, 0x1000, 0x40001b9b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400007e480, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40001a2870) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40001a2870, 0x400048e000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40001615c0, 0x400048e000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40001615c0, 0x400048e000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000161600, 0x400048e000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x400016e2c0, 0x0, 0x40001b9eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400016e2c0, 0x0, 0x0, 0x40001b9f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400016e2c0, 0xee0d40, 0x400009c280, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40000cf5f0, 0x4000138e40, 0x4000100360) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 44 [select]: net/http.(*persistConn).writeLoop(0x400014e6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 90 [select]: net/http.(*persistConn).readLoop(0x40004b4900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 91 [select]: net/http.(*persistConn).writeLoop(0x40004b4900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 50 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000099540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 46 [select]: net/http.(*persistConn).writeLoop(0x400000d0e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 49 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x400441f300, 0x4000262af0, 0x4000262b60, 0x4004c56350, 0x12e8f90, 0x40004fcb60, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x4004c26024, 0x5, 0x400441eec0, 0x12eca70, 0x400441ed80, 0x130d478, 0x4000180310, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x4004472540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x4004472540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 48 [IO wait]: internal/poll.runtime_pollWait(0xffff97c2fb90, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400443a598, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400443a580, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400443a580, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c1e348, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c1e348, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x400441eec0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 52 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x40024a6380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 47 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400049f700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 45 [select]: net/http.(*persistConn).readLoop(0x400000d0e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 55 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x40024a6380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 56 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x40024a6380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 57 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x40024a6380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 68 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x400007ecc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 69 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x40025469c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 70 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x40000aa680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 71 [IO wait]: internal/poll.runtime_pollWait(0xffff97c2fe48, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000aac98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000aac80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000aac80, 0x680da0, 0x4000058698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40001887b0, 0x4000038900, 0x400007e300, 0x4000001d08) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40001887b0, 0x12ed9d0, 0x40001887b0, 0x58ec48, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x40000aac00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 72 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400252ab60, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 73 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400252ab60, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 74 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x400252ab60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 75 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x40025474a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 81 [IO wait]: internal/poll.runtime_pollWait(0xffff97c2faa8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000ab198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000ab180, 0x40001af000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000ab180, 0x40001af000, 0x1000, 0x1000, 0x40044e9688, 0x69f178, 0x40044e96f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000101a8, 0x40001af000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000cc60, 0x40001af000, 0x1000, 0x1000, 0x5, 0x4000410000, 0x40044e97c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002426ae0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002426ae0, 0x68f90a, 0x400038bce0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4002426ae0, 0x0, 0x40044e9868, 0x41a0a8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40001a3e90) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40001a3e90, 0x40044f4000, 0x1000, 0x1000, 0x40044e9998, 0x44e884, 0x1c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000098340, 0x40044f4000, 0x1000, 0x1000, 0x40001396d8, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000098340, 0x40044f4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000098380, 0x40044f4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40044e9d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40044e9d80, 0xa, 0x40044f4201, 0x1, 0xffff97c7c2a0, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40044e9d80, 0x40044f420a, 0x1, 0x4004414280, 0x0, 0x40044f4259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40044e9d80, 0x400241220a, 0x4004414280, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000b13c0, 0x400247f8e0, 0x1f, 0x40044e9ee0, 0x4000188ca8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x40000b13c0, 0x113075a, 0x7, 0x400240e620, 0xd, 0x4000188ca8, 0x0, 0x0, 0x2) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x40000b1380, 0x400240e620, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 79 [IO wait]: internal/poll.runtime_pollWait(0xffff97c2fd60, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000aae18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000aae00, 0x400019f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000aae00, 0x400019f000, 0x1000, 0x1000, 0x4000065be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010140, 0x400019f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c6c0, 0x400019f000, 0x1000, 0x1000, 0x4000065cf8, 0x67867c, 0x4000039860) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400007fb60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x400007fb60, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000c6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 78 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40003e4690) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 80 [select]: net/http.(*persistConn).writeLoop(0x400000c6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 30 [IO wait]: internal/poll.runtime_pollWait(0xffff97c2fc78, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000250c18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000250c00, 0x40044e3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000250c00, 0x40044e3000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000a81a0, 0x40044e3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40004b46c0, 0x40044e3000, 0x1000, 0x1000, 0x4000105500, 0x200000003, 0x4000105500) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000139b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000139b00, 0x400025c00a, 0x40024578c8, 0x4d3b10, 0x4000138060, 0x400025c000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000139b00, 0x0, 0x1, 0x0, 0x4000138078, 0x400025c000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4002547b30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4002547b30, 0x40001a7000, 0x1000, 0x1000, 0x40024579b8, 0x0, 0x4002457988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40002028c0, 0x40001a7000, 0x1000, 0x1000, 0x4000001e00, 0x3, 0x4002457be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40002028c0, 0x40001a7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000202900, 0x40001a7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4002457dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002457dd0, 0xa, 0x40001a7201, 0x1, 0xd1f, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4002457dd0, 0x40001a720a, 0x1, 0x400240e609, 0x0, 0x40001a72e1, 0x1, 0xd1f, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4002457dd0, 0x400252ad0a, 0x400240e609, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000b13c0, 0x40000ae810, 0x17, 0x4002457f20, 0x40044d2858, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x40000b13c0, 0x113075a, 0x7, 0x40044d2858, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x40000b1380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 31 [IO wait]: internal/poll.runtime_pollWait(0xffff97c2ff30, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000250818, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000250800, 0x40001e8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000250800, 0x40001e8000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000a8080, 0x40001e8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000d0e0, 0x40001e8000, 0x1000, 0x1000, 0x4000105680, 0x200000003, 0x4000105680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001391a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001391a0, 0x400025c00a, 0x40024558c8, 0x4d3b10, 0x4000138060, 0x400025c000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40001391a0, 0x0, 0x1, 0x0, 0x4000138078, 0x400025c000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40001a3ad0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40001a3ad0, 0x40044f1000, 0x1000, 0x1000, 0x40024559b8, 0x44d8a8, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000161f00, 0x40044f1000, 0x1000, 0x1000, 0x40001396d8, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000161f00, 0x40044f1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000161f40, 0x40044f1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4002455dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002455dd0, 0xa, 0x40044f1201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4002455dd0, 0x40044f120a, 0x1, 0x40000a006b, 0x0, 0x40044f1263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4002455dd0, 0x40000a670a, 0x40000a006b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40000b13c0, 0x40000b2d50, 0x24, 0x4002455f20, 0x40044d2840, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x40000b13c0, 0x113075a, 0x7, 0x40044d2840, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x40000b1380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 92 [IO wait]: internal/poll.runtime_pollWait(0xffff97c30018, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000250f18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000250f00, 0x40044f2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000250f00, 0x40044f2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000a81e0, 0x40044f2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40004b4900, 0x40044f2000, 0x1000, 0x1000, 0x20, 0x30, 0x20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400040f0e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400040f0e0, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400040f0e0, 0x7356e4, 0xffffe71b294e, 0xe, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000201350) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000201350, 0x4004c8a400, 0x200, 0x200, 0xffffe71b294e, 0xe, 0x4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000099a80, 0x4004c8a400, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000099a80, 0x4004c8a400, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000099ac0, 0x4004c8a400, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004c82f20, 0x0, 0x4004582eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004c82f20, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004c82f20, 0xee0d40, 0x400009ce60, 0x12db068, 0x4000099ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40024cb320, 0x4004c804e0, 0x40024fe7e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 82 [select]: net/http.(*persistConn).readLoop(0x40004b46c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 83 [select]: net/http.(*persistConn).writeLoop(0x40004b46c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 61 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40000a2760, 0x1144559, 0x1b, 0x4000489e38, 0x4004609ce0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x40046bcec0, 0x4004609ce0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000489f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40046da900, 0x40046bcec0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 60 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x4004436820, 0x4004609bc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 102 [select]: net/http.(*persistConn).readLoop(0x400000cc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 103 [select]: net/http.(*persistConn).writeLoop(0x400000cc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 58 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000098100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 59 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400049ef00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 115 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40004fcb60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 116 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x4002418a20, 0x4004421b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 117 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x4000098040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 118 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x4004436820) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 119 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x4004436820) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 120 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x4000136b18) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 121 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x40044368c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 122 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x40044368c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 123 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x400441f300, 0x4004c56350) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 62 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 63 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x40004acaf0, 0xd00007, 0x40046fd388, 0x5, 0x40046bced0, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 64 [IO wait]: internal/poll.runtime_pollWait(0xffff97c2f9c0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000191698, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000191680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000191680, 0xffff97ce7f18, 0x30, 0x40046b7560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004bde7b0, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004bde7b0, 0x40046b7560, 0x11ae500, 0x40004fd040, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40004fcfc0, 0x12ed9d0, 0x4004bde7b0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40004fcfc0, 0x12ed9d0, 0x4004bde7b0, 0x4000214000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 65 [IO wait]: internal/poll.runtime_pollWait(0xffff97c2f8d8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000191798, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000191780, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000191780, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004bde7c8, 0xffff97ce7f18, 0xffff4fd04130, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004bde7c8, 0x665e1c, 0x0, 0xfe5e60, 0x40001c75c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004bde7e0, 0x40046b75f0, 0x11ae500, 0x40004fd120, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40004fd0a0, 0x12ec710, 0x4004bde7e0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40004fd0a0, 0x12ec710, 0x4004bde7e0, 0x4000214000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 148 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40004fd340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 136 [select]: net/http.(*persistConn).writeLoop(0x40000fe000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 130 [IO wait]: internal/poll.runtime_pollWait(0xffff97c410c0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400443a998, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400443a980, 0x4004c25000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400443a980, 0x4004c25000, 0x1000, 0x1000, 0x4004c74be0, 0x469b10, 0x4004c62f00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000136ba8, 0x4004c25000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4002411b00, 0x4004c25000, 0x1000, 0x1000, 0x4004c74cf8, 0x67867c, 0x4004db02a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004471560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004471560, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4002411b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 131 [select]: net/http.(*persistConn).writeLoop(0x4002411b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 149 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40004fd340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 147 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40004fd340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 146 [semacquire]: sync.runtime_Semacquire(0x4004cd10d8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004cd10d0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x40000df080, 0x40001a2360, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffe71b2969, 0x29, 0xffffe71b29cb, 0x25, 0x40024ff740, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004c26d10, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffe71b2969, 0x29, 0xffffe71b29cb, 0x25, 0x40024ff740, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004c26d10, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004c91190, 0x4004c90e60, 0x40001a2360, 0x400026b680, 0x4000099540, 0x0, 0x1c211e8, 0x40044e6d98, 0x412504, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x4000214000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004c26d10, 0xb, 0x4004c26d08, 0x6, 0x4004c26d20, 0xd, 0x4004c1abe0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x4000214000, 0x4000400000, 0x40002000f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x4000214000, 0x40024f8ab0, 0x2, 0x2, 0x40024f8ac0, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 150 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40004fd340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 151 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4004bec240) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 137 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).processSchedIndexes(0x4004b8a630) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:296 +0x44 created by github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:218 +0x3c goroutine 135 [IO wait]: internal/poll.runtime_pollWait(0xffff97c411a8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000191d18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000191d00, 0x4004e19000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000191d00, 0x4004e19000, 0x1000, 0x1000, 0x4004c6ebe0, 0x469b10, 0x400252c300) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000136c58, 0x4004e19000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40000fe000, 0x4004e19000, 0x1000, 0x1000, 0x4004c6ecf8, 0x67867c, 0x4004db1380) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004471e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004471e00, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40000fe000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 154 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004b8a790, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 155 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004b8a790, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 156 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4004b8a790) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 157 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40004fd340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 158 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40000a2760, 0x1144559, 0x1b, 0x4004e87e38, 0x4004bee360, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x40046bd360, 0x4004bee360, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004e87f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40046daf60, 0x40046bd360) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 161 [select]: net/http.(*persistConn).readLoop(0x40000fe120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 162 [select]: net/http.(*persistConn).writeLoop(0x40000fe120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 178 [IO wait]: internal/poll.runtime_pollWait(0xffff97c40fd8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000191e98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000191e80, 0x4004bfa000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000191e80, 0x4004bfa000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40004aa350, 0x4004bfa000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40000fe120, 0x4004bfa000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40046dbc20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40046dbc20, 0x100a, 0x0, 0x0, 0x40001c93e8, 0x597c30, 0x40001c9438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40046dbc20, 0x0, 0x40001c9478, 0x655784, 0x40001c94a8, 0x40b2f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000201a40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000201a40, 0x4004c8ae00, 0x200, 0x200, 0x4, 0x0, 0x40001c9508) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000099d40, 0x4004c8ae00, 0x200, 0x200, 0x4004c881c7, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000099d40, 0x4004c8ae00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000099d80, 0x4004c8ae00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004c83b80, 0x680d00, 0x40001c96b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004c83b80, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004c83b80, 0xee0d40, 0x400009cff0, 0x12db068, 0x4000099d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40024cb7a0, 0x4004c80c60, 0x40024ff080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 179 [runnable]: time.Now(0x40046dd020, 0x1b, 0x40046dfd40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/time/time.go:1066 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x40046dd020, 0x1b, 0x4004c79580, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:519 +0x78 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x40046dfcb0, 0x112f905, 0x6, 0x12db908, 0x40004a3240, 0xee40c0, 0x4004eb0098, 0x4004c79580, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x4000130480, 0x46, 0x12db908, 0x40004a3240, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x4000130480, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x20, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004c79d40, 0x4, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x40000fd680, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x40000fd680, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4004bec280) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1267 +0xf8 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 180 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x4004b8a630) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:224 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 181 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x4004b8a630) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:612 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 182 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x4004b8a630) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:662 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 183 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x4004b8a630) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 184 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x40024f3500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:249 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 186 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x4000214000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 187 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x4000214000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1090 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 188 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x4000214000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 189 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 goroutine 138 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).registerWithServer(0x40024f3500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:296 created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:205 +0x78 goroutine 139 [runnable]: strings.ToLower(0x4004e380c0, 0x4, 0x4004e380c0, 0x4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/strings/strings.go:580 +0x26c net/url.parse(0x4004e380c0, 0x26, 0x4004e30100, 0x4004e380c0, 0x26, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/url/url.go:531 +0x130 net/url.Parse(0x4004e380c0, 0x26, 0x7, 0x4004e37540, 0x1f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/url/url.go:478 +0x70 github.com/couchbase/indexing/secondary/security.GetURL(0x4004e37540, 0x1f, 0x4004580c48, 0x456974, 0x1f) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:392 +0x40 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004e37540, 0x1f, 0x4004c71d10, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:514 +0x5c github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/indexer.getWithAuth(0x4004e37540, 0x1f, 0xe, 0x113846e, 0x11) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:3077 +0x68 github.com/couchbase/indexing/secondary/indexer.getGlobalTopology(0x4004cd06f0, 0xe, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:2266 +0x78 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).updateNodeList(0x40024f3500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:220 +0x3c created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:206 +0x98 Initializing write barrier = 8000 2021-08-16T13:51:39.154+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:51:39.154+00:00 [Info] Setting ipv6=false 2021-08-16T13:51:39.168+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:51:39.175+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:51:39.175+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:51:39.175+00:00 [Info] Setting log level to Info 2021-08-16T13:51:39.175+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:51:39.178+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021-08-16T13:51:39.186+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:51:39.186+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:51:39.187+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:51:39.187+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:51:39.187+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021/08/16 13:51:39 audit: created new audit service 2021-08-16T13:51:39.198+00:00 [Info] security context initialized 2021-08-16T13:51:39.198+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:51:39.199+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:51:39.199+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:51:39.199+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:51:39.199+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:51:39.212+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:51:39.214+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:51:39.226+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:51:39.239+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:51:39.239+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:51:39.241+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:51:39.242+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:51:39.248+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:51:39.251+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:51:39.251+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:51:39.257+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:51:39.302+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:51:39.302+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 50 us 2021-08-16T13:51:39.302+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:51:39.305+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:51:39.305+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:51:39.306+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:51:39.306+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:51:39.306+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:51:39.306+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:51:39.306+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:51:39.313+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:39.313+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:39.315+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:51:39.315+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:51:39.315+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:51:39.315+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:39.315+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:51:39.315+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:51:39.315+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:51:39.315+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:51:39.315+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:51:39.315+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:51:39.315+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:51:39.319+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 3.797395ms 2021-08-16T13:51:39.319+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:51:39.320+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:51:39.320+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:39.320+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:51:39.320+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:51:39.320+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:51:39.320+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... unexpected fault address 0x40043cf21f fatal error: fault 2021-08-16T13:51:39.331+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:51:39.331+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:51:39.332+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:51:39.332+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:51:39.332+00:00 [Info] : Recover shard shard1 from metadata completed. [signal SIGBUS: bus error code=0x1 addr=0x40043cf21f pc=0xbd8ea4] goroutine 183 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004cf09f0 sp=0x4004cf09c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004cf0a30 sp=0x4004cf09f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x4004778820, 0x40046d9800, 0x11a9890, 0x4004cf0b40, 0x4004778838, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004cf0ab0 sp=0x4004cf0a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x4004778820, 0x40046d9800, 0x11a9890, 0x4004cf0b40, 0x4004778838, 0xffffa8293f60, 0x4004cf0c18, 0x49) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004cf0af0 sp=0x4004cf0ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x400237eb40, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004cf0da0 sp=0x4004cf0af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x400237eb40, 0x40001fb490, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004cf0e60 sp=0x4004cf0da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004e6e080, 0x4004e6e090, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004cf1200 sp=0x4004cf0e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004e6e080, 0x4004e6e090, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004cf18f0 sp=0x4004cf1200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004e56c50, 0x4004e60700, 0x400245db80, 0x4004e6e060) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004cf1fb0 sp=0x4004cf18f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004cf1fb0 sp=0x4004cf1fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x40000e2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x40001129c0, 0x4, 0x1136a87, 0xf, 0x40000f9e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 7 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400000e690) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 8 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400000e6a8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 9 [IO wait]: internal/poll.runtime_pollWait(0xffff803c5198, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0898, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0880, 0x40001a3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0880, 0x40001a3000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000105b0, 0x40001a3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x4000082cc0, 0x40001a8001, 0xdff, 0xdff, 0x400009f8e8, 0x6902d8, 0x40000f5948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400000e6f0, 0x40001a8001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40000ce9a0, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40000ce9a0, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40000ce9a0, 0xf5a960, 0x4000082da0, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4000082d80, 0x40000467c0, 0x40000f5b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400008c820, 0x12f8f98, 0x4000082d80, 0x4000037447, 0x16, 0x400000d440, 0x400009fc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400008c820, 0x12f8f98, 0x4000082d80, 0x400008c820, 0x4000036c58, 0x4000036c60, 0x40000f0900, 0x40000466e0, 0xee0980, 0x400000d440, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400008c820, 0x12f8f98, 0x4000082d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x40000fa3c0, 0x400009ff78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400009df78, 0x40000fa3c0, 0x12db8a8, 0x400000e6c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x40000fa3c0, 0x40000b1540, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x40000fa3c0, 0x40000b1540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 11 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400000ac00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 18 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 19 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 20 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 21 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 22 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 23 [syscall]: syscall.Syscall(0x3f, 0x0, 0x4000054fa0, 0x4, 0x4000054e68, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x4000054fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x4000054fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x4000054fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 37 [select]: net/http.(*persistConn).writeLoop(0x400000cb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 34 [select]: net/http.(*persistConn).readLoop(0x400000c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 35 [select]: net/http.(*persistConn).writeLoop(0x400000c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 27 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400010e2a0, 0x12c18bc, 0x1, 0x4000017e58, 0x4000038900, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x40001c36c8, 0x4000038900, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000017fa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4000080000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 28 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4000080000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 44 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x40001ebac0, 0x40003cad20, 0x40003cad90, 0x4004c02d40, 0x12e8f90, 0x400237c9a0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x40047fa9b4, 0x5, 0x40001eb680, 0x12eca70, 0x40001eb500, 0x130d478, 0x400019a150, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40000e8b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40000e8b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 36 [IO wait]: internal/poll.runtime_pollWait(0xffff803c4fc8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0280, 0x40013a2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0280, 0x40013a2000, 0x1000, 0x1000, 0x400006abe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001240c8, 0x40013a2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000cb40, 0x40013a2000, 0x1000, 0x1000, 0x400006acf8, 0x67867c, 0x4000136300) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000348660) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000348660, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000cb40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 15 [IO wait]: internal/poll.runtime_pollWait(0xffff803c50b0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0180, 0x40001a9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0180, 0x40001a9000, 0x1000, 0x1000, 0x40003fba88, 0x68f7a4, 0x40001ee050) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001240c0, 0x40001a9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c5a0, 0x40001a9000, 0x1000, 0x1000, 0xf12180, 0x400037c180, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000348420) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000348420, 0xa, 0x1008880, 0x400037c140, 0x199, 0x1000, 0x40003fbb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000348420, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000096d50) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000096d50, 0x40001f2000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40001ea000, 0x40001f2000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40001ea000, 0x40001f2000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001ea040, 0x40001f2000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40001ee000, 0x0, 0x40003fbeb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40001ee000, 0x0, 0x0, 0x40003fbf70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40001ee000, 0xee0d40, 0x400037c140, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400011d050, 0x40001e0120, 0x40001ec000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 85 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x40000e8620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 57 [IO wait]: internal/poll.runtime_pollWait(0xffff803c47a0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0618, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0600, 0x40044f3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0600, 0x40044f3000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010550, 0x40044f3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000114ea0, 0x40044f3000, 0x1000, 0x1000, 0x40001b5500, 0x200000003, 0x40001b5500) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002352d20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002352d20, 0x400001c00a, 0x40000198c8, 0x4d3b10, 0x40000820c0, 0x400001c000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4002352d20, 0x0, 0x1, 0x0, 0x40000820d8, 0x400001c000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40000fb6e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40000fb6e0, 0x4000330000, 0x1000, 0x1000, 0x40000199e8, 0x44e884, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400013e9c0, 0x4000330000, 0x1000, 0x1000, 0x4000348e98, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400013e9c0, 0x4000330000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400013ea00, 0x4000330000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000019dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000019dd0, 0xa, 0x4000330201, 0x1, 0xffffa8295da8, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000019dd0, 0x400033020a, 0x1, 0x40001ab990, 0x0, 0x4000330263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000019dd0, 0x400004670a, 0x40001ab990, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000126ad0, 0x400037a810, 0x24, 0x4000019f20, 0x40023d5428, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x4000126ad0, 0x113075a, 0x7, 0x40023d5428, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x4000126a90) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 56 [IO wait]: internal/poll.runtime_pollWait(0xffff803c4970, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40043ca198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40043ca180, 0x400249a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40043ca180, 0x400249a000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000109e8, 0x400249a000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40043d0000, 0x400249a000, 0x1000, 0x1000, 0x40001b5680, 0x200000003, 0x40001b5680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002353320) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002353320, 0x400001c00a, 0x40013f18c8, 0x4d3b10, 0x40000820c0, 0x400001c000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4002353320, 0x0, 0x1, 0x0, 0x40000820d8, 0x400001c000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000097f50) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000097f50, 0x40003c5000, 0x1000, 0x1000, 0x40013f19b8, 0x0, 0x40013f1988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40043b0880, 0x40003c5000, 0x1000, 0x1000, 0x4000102600, 0x3, 0x40013f1be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40043b0880, 0x40003c5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40043b08c0, 0x40003c5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40013f1dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40013f1dd0, 0xa, 0x40003c5201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40013f1dd0, 0x40003c520a, 0x1, 0x400439d929, 0x0, 0x40003c52e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40013f1dd0, 0x40023db20a, 0x400439d929, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000126ad0, 0x400017a7b0, 0x17, 0x40013f1f20, 0x40013e43d8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x4000126ad0, 0x113075a, 0x7, 0x40013e43d8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x4000126a90) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 16 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400234a5c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 55 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x400011a930) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 43 [IO wait]: internal/poll.runtime_pollWait(0xffff803c4888, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000141918, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000141900, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000141900, 0x58ec48, 0x4000039380, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c044c8, 0x4714f4, 0x4000039380, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c044c8, 0x40001e3d00, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x40001eb680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 42 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40043b1b40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 87 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x40000e8620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 88 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x40000e8620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 89 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x40000e8620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 53 [select]: net/http.(*persistConn).readLoop(0x40043d0000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 54 [select]: net/http.(*persistConn).writeLoop(0x40043d0000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 67 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40043b0080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 68 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x40043b6de0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 69 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x40043a09f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 70 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x40043ca280) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 71 [IO wait]: internal/poll.runtime_pollWait(0xffff803c4a58, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40043ca398, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40043ca380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40043ca380, 0x0, 0x40001c5698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40001d6240, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40001d6240, 0x12ed9d0, 0x40001d6240, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x40043ca300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 72 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40023da410, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 73 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40023da410, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 74 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x40023da410) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 75 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x40043a14d0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 91 [select]: net/http.(*persistConn).writeLoop(0x4000114ea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 127 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40000e90a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 95 [select]: net/http.(*persistConn).writeLoop(0x40023d8c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 94 [select]: net/http.(*persistConn).readLoop(0x40023d8c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 104 [IO wait]: internal/poll.runtime_pollWait(0xffff803c45d0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000141198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000141180, 0x40013f2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000141180, 0x40013f2000, 0x1000, 0x1000, 0x400009f688, 0x69f178, 0x400009f6f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001fe128, 0x40013f2000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000cc60, 0x40013f2000, 0x1000, 0x1000, 0x5, 0x4002400000, 0x400009f7c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001e0e40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001e0e40, 0x68f90a, 0x40001f49a0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40001e0e40, 0x0, 0x400009f868, 0x41a0a8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40001a6990) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40001a6990, 0x40003d3000, 0x1000, 0x1000, 0x400009f968, 0x44d8a8, 0x1c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40043b0a80, 0x40003d3000, 0x1000, 0x1000, 0x4000082bf8, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40043b0a80, 0x40003d3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40043b0ac0, 0x40003d3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400009fd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400009fd80, 0xa, 0x40003d3201, 0x1, 0xda7, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400009fd80, 0x40003d320a, 0x1, 0x40001e25d8, 0x0, 0x40003d3259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400009fd80, 0x40003ec50a, 0x40001e25d8, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000126ad0, 0x40013e3c00, 0x1f, 0x400009fee0, 0x40013e43f0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x4000126ad0, 0x113075a, 0x7, 0x400439d940, 0xd, 0x40013e43f0, 0x0, 0x0, 0x2) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x4000126a90, 0x400439d940, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 113 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400234b300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 97 [IO wait]: internal/poll.runtime_pollWait(0xffff803c4400, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40044f0498, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40044f0480, 0x400438b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40044f0480, 0x400438b000, 0x1000, 0x1000, 0x400237ba48, 0x652ce0, 0x12edac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010588, 0x400438b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40023d8c60, 0x400438b000, 0x1000, 0x1000, 0x113674e, 0x412bc0, 0x40043c4000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002353980) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002353980, 0x100a, 0x0, 0x0, 0x0, 0x0, 0x400247bc38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4002353980, 0x0, 0x400247bc78, 0x655784, 0x400247bca8, 0x40b2f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40045c5650) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40045c5650, 0x4004bf7400, 0x200, 0x200, 0x60, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400234bd80, 0x4004bf7400, 0x200, 0x200, 0x40023ede60, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400234bd80, 0x4004bf7400, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400234bdc0, 0x4004bf7400, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004bfd760, 0x680d00, 0x400247beb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004bfd760, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004bfd760, 0xee0d40, 0x40043c48c0, 0x12db068, 0x400234bdc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004bfa990, 0x40045d2a80, 0x4004cb80c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 96 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40043b06c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 40 [select]: net/http.(*persistConn).readLoop(0x400000cc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 41 [select]: net/http.(*persistConn).writeLoop(0x400000cc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 90 [select]: net/http.(*persistConn).readLoop(0x4000114ea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 111 [IO wait]: internal/poll.runtime_pollWait(0xffff803c44e8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40043cab18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40043cab00, 0x4004516000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40043cab00, 0x4004516000, 0x1000, 0x1000, 0x4000350be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40043ae168, 0x4004516000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40043d10e0, 0x4004516000, 0x1000, 0x1000, 0x4000350cf8, 0x67867c, 0x40000393e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000083920) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000083920, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40043d10e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 112 [select]: net/http.(*persistConn).writeLoop(0x40043d10e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 45 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x400237c9a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 46 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x400237c9a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 47 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x40000de900, 0x40047a0de0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 48 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x40043b0680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 49 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x4004c08280) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 114 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x4004c08280) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 115 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x40001fe520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 116 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x4004c08320) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 117 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x4004c08320) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 118 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x40001ebac0, 0x4004c02d40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 119 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x4004c08280, 0x40047a0f00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 120 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400010e2a0, 0x1144559, 0x1b, 0x4004585e38, 0x40047a1020, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004c03520, 0x40047a1020, 0xf1f700, 0x400010fb20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004585f80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004c289c0, 0x4004c03520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 121 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 122 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400037cc30, 0xd00007, 0x4004c4c4f8, 0x5, 0x4004c03530, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 123 [IO wait]: internal/poll.runtime_pollWait(0xffff803c46b8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000141b18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000141b00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000141b00, 0xffffa828aa68, 0x30, 0x4004d64510) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c04a80, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c04a80, 0x4004d64510, 0x11ae500, 0x40000e8cc0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40000e8c40, 0x12ed9d0, 0x4004c04a80, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40000e8c40, 0x12ed9d0, 0x4004c04a80, 0x40000e2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 124 [IO wait]: internal/poll.runtime_pollWait(0xffff803c4318, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000141c18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000141c00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000141c00, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c04a98, 0xffffa828aa68, 0xffff800911e8, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c04a98, 0x665e1c, 0x0, 0xfe5e60, 0x4004c52dc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004c04ab0, 0x4004d645a0, 0x11ae500, 0x40000e8da0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40000e8d20, 0x12ec710, 0x4004c04ab0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40000e8d20, 0x12ec710, 0x4004c04ab0, 0x40000e2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 147 [IO wait]: internal/poll.runtime_pollWait(0xffff800c1fc0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40043cb498, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40043cb480, 0x400477e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40043cb480, 0x400477e000, 0x1000, 0x1000, 0x4000351be0, 0x469b10, 0x40043c6300) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40043ae290, 0x400477e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40023d97a0, 0x400477e000, 0x1000, 0x1000, 0x4000351cf8, 0x67867c, 0x4004531440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004554e40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004554e40, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40023d97a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 148 [select]: net/http.(*persistConn).writeLoop(0x40023d97a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 126 [semacquire]: sync.runtime_Semacquire(0x4004e56c58) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004e56c50) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x400245db80, 0x40000fb620, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffe8e20969, 0x29, 0xffffe8e209cb, 0x25, 0x4004e1a5a0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x40047fb750, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffe8e20969, 0x29, 0xffffe8e209cb, 0x25, 0x4004e1a5a0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x40047fb750, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004cf1190, 0x4004cf0e60, 0x40000fb620, 0x40000b8b40, 0x400234a5c0, 0x0, 0xbf, 0x40013eeda8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x40000e2000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x40047fb750, 0xb, 0x40047fb748, 0x6, 0x40047fb760, 0xd, 0x4004c008c0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x40000e2000, 0x4000340000, 0x40000fa4b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x40000e2000, 0x40043a8970, 0x2, 0x2, 0x40043a8980, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 156 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000081880) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 135 [IO wait]: internal/poll.runtime_pollWait(0xffff800c20a8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004dc0298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004dc0280, 0x4004dde000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004dc0280, 0x4004dde000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001fe678, 0x4004dde000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004c2efc0, 0x4004dde000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004c29f20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004c29f20, 0xa, 0x0, 0x0, 0x0, 0x0, 0x40001c0c38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004c29f20, 0x0, 0x40001c0c78, 0x655784, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004cf8750) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004cf8750, 0x4004e90400, 0x200, 0x200, 0x4, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004cfa240, 0x4004e90400, 0x200, 0x200, 0x40045cc087, 0x33, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004cfa240, 0x4004e90400, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004cfa280, 0x4004e90400, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004cfd340, 0x0, 0x40001c0eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004cfd340, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004cfd340, 0xee0d40, 0x40043c4ff0, 0x12db068, 0x4004cfa280) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004bfb3b0, 0x40045d3440, 0x4004cb8ea0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 128 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40000e90a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 129 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40000e90a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 162 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40000e90a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 163 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x40000817c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 164 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004c0ac60, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 165 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004c0ac60, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 166 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4004c0ac60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 167 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40000e90a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 168 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400010e2a0, 0x1144559, 0x1b, 0x4004ddde38, 0x40047a1b00, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004c03cb0, 0x40047a1b00, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004dddf80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004c293e0, 0x4004c03cb0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 171 [select]: net/http.(*persistConn).readLoop(0x4004c2efc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 172 [select]: net/http.(*persistConn).writeLoop(0x4004c2efc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 157 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x4004c0ab00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 158 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x4004c0ab00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 159 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x4004c0ab00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 160 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x4004c0ab00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 161 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x40046c9600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 178 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x40046c9600, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 179 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x40000e2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 180 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x40000e2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 181 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x40000e2000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 182 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:51:44.482+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:51:44.482+00:00 [Info] Setting ipv6=false 2021-08-16T13:51:44.499+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:51:44.504+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:51:44.504+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:51:44.504+00:00 [Info] Setting log level to Info 2021-08-16T13:51:44.504+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:51:44.507+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:51:44 audit: created new audit service 2021-08-16T13:51:44.515+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:51:44.516+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:51:44.516+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:51:44.516+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:51:44.516+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:51:44.527+00:00 [Info] security context initialized 2021-08-16T13:51:44.527+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:51:44.527+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:51:44.527+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:51:44.527+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:51:44.527+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:51:44.542+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:51:44.547+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:51:44.547+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:51:44.562+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:51:44.564+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:51:44.565+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:51:44.565+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:51:44.580+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:51:44.580+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:51:44.584+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:51:44.586+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:51:44.628+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 50 us 2021-08-16T13:51:44.628+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:51:44.628+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:51:44.630+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:51:44.631+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:51:44.631+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:51:44.631+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:51:44.631+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:51:44.631+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:51:44.631+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:51:44.632+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:44.633+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:44.634+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:51:44.634+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:51:44.634+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:51:44.634+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:44.635+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:51:44.635+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:51:44.635+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:51:44.635+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:51:44.635+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:51:44.635+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:51:44.635+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:51:44.641+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 6.786009ms 2021-08-16T13:51:44.642+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:51:44.643+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:51:44.643+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:44.643+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:51:44.643+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:51:44.643+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:51:44.643+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:51:44.654+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:51:44.654+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:51:44.654+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:51:44.654+00:00 [Info] RebalanceMgr::initService Init 2021-08-16T13:51:44.655+00:00 [Info] ServiceMgr::registerWithServer nodeuuid 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:44.655+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:51:44.655+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x4004766b5f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x4004766b5f pc=0xbd8ea4] goroutine 200 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004ec29f0 sp=0x4004ec29c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004ec2a30 sp=0x4004ec29f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x400472cd00, 0x400343d800, 0x11a9890, 0x4004ec2b40, 0x400472cd18, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004ec2ab0 sp=0x4004ec2a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x400472cd00, 0x400343d800, 0x11a9890, 0x4004ec2b40, 0x400472cd18, 0xc18f8c, 0x4004ec2c18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004ec2af0 sp=0x4004ec2ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x40043eeb40, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004ec2da0 sp=0x4004ec2af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x40043eeb40, 0x4004eba150, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004ec2e60 sp=0x4004ec2da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x40047596b0, 0x40047596c0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004ec3200 sp=0x4004ec2e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x40047596b0, 0x40047596c0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004ec38f0 sp=0x4004ec3200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004dd5b50, 0x4004ea8e00, 0x40043e9b80, 0x4004759690) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004ec3fb0 sp=0x4004ec38f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004ec3fb0 sp=0x4004ec3fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x40003a0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x40000fb440, 0x4, 0x1136a87, 0xf, 0x40000f9e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 7 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400000e690) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 8 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400000e6a8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 9 [IO wait]: internal/poll.runtime_pollWait(0xffff7d386848, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0e18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0e00, 0x40013c1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0e00, 0x40013c1000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40013a4000, 0x40013c1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x40013ce000, 0x40003a6001, 0xdff, 0xdff, 0x400009b8e8, 0x6902d8, 0x40000f9948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x40013cc018, 0x40003a6001, 0xdff, 0xdff, 0x40000f99f8, 0x6f1968, 0x40000f9938) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40000cef20, 0xa, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40000cef20, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40000cef20, 0xf5a960, 0x400032d760, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x400032d740, 0x4000046920, 0x40000f9b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400008cbe0, 0x12f8f98, 0x400032d740, 0x40000377a7, 0x16, 0x400031b9e0, 0x400009bc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400008cbe0, 0x12f8f98, 0x400032d740, 0x400008cbe0, 0x4000036fb8, 0x4000036fc0, 0x40000f0e80, 0x4000046840, 0xee0980, 0x400031b9e0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400008cbe0, 0x12f8f98, 0x400032d740) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x40000fa3c0, 0x400009bf78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400009bf78, 0x40000fa3c0, 0x12db8a8, 0x400000ed20, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x40000fa3c0, 0x40000b1180, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x40000fa3c0, 0x40000b1180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 10 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 11 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 12 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 13 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 14 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 15 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40000547a0, 0x4, 0x4000054668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 34 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400000ac00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 40 [IO wait]: internal/poll.runtime_pollWait(0xffff7d385e50, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1a98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f1a80, 0x4004510000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f1a80, 0x4004510000, 0x1000, 0x1000, 0x1, 0x0, 0x40000a16f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010c68, 0x4004510000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003b2b40, 0x4004510000, 0x1000, 0x1000, 0x5, 0x40003d2a00, 0x40000a17c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002468900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002468900, 0x68f90a, 0x40024389a0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4002468900, 0x0, 0x2, 0x0, 0x1, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400246e4e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400246e4e0, 0x4004514000, 0x1000, 0x1000, 0x40000a1998, 0x44e884, 0x40000a1a18) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40013c38c0, 0x4004514000, 0x1000, 0x1000, 0x40013cf558, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40013c38c0, 0x4004514000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40013c3900, 0x4004514000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40000a1d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000a1d80, 0xa, 0x4004514201, 0x1, 0xffffa7a39658, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40000a1d80, 0x400451420a, 0x1, 0x4000036960, 0x0, 0x4004514259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40000a1d80, 0x400244380a, 0x4000036960, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40023f8520, 0x4002462b60, 0x1f, 0x40000a1ee0, 0x400247e270, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x40023f8520, 0x113075a, 0x7, 0x400245cd00, 0xd, 0x400247e270, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x40023f84e0, 0x400245cd00, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 54 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001943c0, 0x12c18bc, 0x1, 0x40013f7e58, 0x4000038120, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x40001146c8, 0x4000038120, 0x40001146c8, 0x679704) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40013f7fa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x40013c2040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 52 [select]: net/http.(*persistConn).readLoop(0x40013b2360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 53 [select]: net/http.(*persistConn).writeLoop(0x40013b2360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 55 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x40013c2040) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 60 [IO wait]: internal/poll.runtime_pollWait(0xffff7d386760, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0180, 0x40003a7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0180, 0x40003a7000, 0x1000, 0x1000, 0x400012ca88, 0x68f7a4, 0x40013beb50) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000100e0, 0x40003a7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40013b2360, 0x40003a7000, 0x1000, 0x1000, 0xf12180, 0x400008c400, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400011c360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400011c360, 0xa, 0x1008880, 0x400008c3c0, 0x199, 0xf90, 0x400012cb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400011c360, 0x2, 0x0, 0x9, 0xa3c, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40013a2a80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40013a2a80, 0x4000019000, 0xe00, 0xe00, 0x400012cda8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40013c2280, 0x4000019000, 0xe00, 0xe00, 0x400012ce08, 0x40bb94, 0x400012ce18) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40013c2280, 0x4000019000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40013c22c0, 0x4000019000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40013beb00, 0x0, 0x400012ceb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40013beb00, 0x0, 0x0, 0x400012cf70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40013beb00, 0xee0d40, 0x400008c3c0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40013ac750, 0x400011cba0, 0x4000038cc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 58 [IO wait]: internal/poll.runtime_pollWait(0xffff7d386678, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0280, 0x4000013000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0280, 0x4000013000, 0x1000, 0x1000, 0x4000069be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010460, 0x4000013000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40013b27e0, 0x4000013000, 0x1000, 0x1000, 0x4000069cf8, 0x67867c, 0x4000038c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400011ca20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x400011ca20, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40013b27e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 59 [select]: net/http.(*persistConn).writeLoop(0x40013b27e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 74 [select]: net/http.(*persistConn).writeLoop(0x400342a480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 63 [select]: net/http.(*persistConn).readLoop(0x40013b3680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 64 [select]: net/http.(*persistConn).writeLoop(0x40013b3680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 21 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40013c3200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 79 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x40033ca7d0, 0xd00007, 0x4004c0aa88, 0x5, 0x4004c14780, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 75 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4003412ac0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 73 [select]: net/http.(*persistConn).readLoop(0x400342a480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 23 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x40000e89a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 24 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x40000e89a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 25 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x40000e89a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 26 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x40000e89a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 71 [select]: net/http.(*persistConn).readLoop(0x400031ac60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 72 [select]: net/http.(*persistConn).writeLoop(0x400031ac60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 27 [IO wait]: internal/poll.runtime_pollWait(0xffff7d386108, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1498, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f1480, 0x400235e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f1480, 0x400235e000, 0x1000, 0x1000, 0x40, 0x1d, 0x40013f37b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000109b8, 0x400235e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40013b3680, 0x400235e000, 0x1000, 0x1000, 0x4001396c00, 0x200000003, 0x4001396c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400011d980) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400011d980, 0x400001a00a, 0x40013f38c8, 0x4d3b10, 0x40000820c0, 0x400001a000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400011d980, 0x0, 0x1, 0x4dd090, 0x40000820d8, 0x400001a000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400246e210) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400246e210, 0x400247c000, 0x1000, 0x1000, 0x40013f39b8, 0x10, 0x40013f3a68) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40013c3780, 0x400247c000, 0x1000, 0x1000, 0x40013cf558, 0x58, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40013c3780, 0x400247c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40013c37c0, 0x400247c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40013f3dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40013f3dd0, 0xa, 0x400247c201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40013f3dd0, 0x400247c20a, 0x1, 0x400245cce9, 0x0, 0x400247c2e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40013f3dd0, 0x40013c7c0a, 0x400245cce9, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40023f8520, 0x400013cb28, 0x17, 0x40013f3f20, 0x4004498048, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x40023f8520, 0x113075a, 0x7, 0x4004498048, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x40023f84e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 28 [IO wait]: internal/poll.runtime_pollWait(0xffff7d386020, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40033cc218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40033cc200, 0x4004440000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40033cc200, 0x4004440000, 0x1000, 0x1000, 0x40, 0x1d, 0x40013f17b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400237e058, 0x4004440000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400031ac60, 0x4004440000, 0x1000, 0x1000, 0x4001396d80, 0x200000003, 0x4001396d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40033e6420) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40033e6420, 0x400001a00a, 0x40013f18c8, 0x4d3b10, 0x40000820c0, 0x400001a000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40033e6420, 0x0, 0x1, 0x31353a3331543631, 0x40000820d8, 0x400001a000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40000979b0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40000979b0, 0x400246b000, 0x1000, 0x1000, 0x40013f19b8, 0x0, 0x40013f1988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40013c3680, 0x400246b000, 0x1000, 0x1000, 0x4001396300, 0x3, 0x40013f1be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40013c3680, 0x400246b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40013c36c0, 0x400246b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40013f1dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40013f1dd0, 0xa, 0x400246b201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40013f1dd0, 0x400246b20a, 0x1, 0x400245ca8b, 0x0, 0x400246b263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40013f1dd0, 0x400004710a, 0x400245ca8b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x40023f8520, 0x40000437a0, 0x24, 0x40013f1f20, 0x4004498030, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x40023f8520, 0x113075a, 0x7, 0x4004498030, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x40023f84e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 30 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x40024694a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 29 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001fcbc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 49 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4004518d00, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 43 [select]: net/http.(*persistConn).readLoop(0x40003b2b40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 44 [select]: net/http.(*persistConn).writeLoop(0x40003b2b40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 47 [IO wait]: internal/poll.runtime_pollWait(0xffff7d385c80, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1c18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f1c00, 0x400453b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f1c00, 0x400453b000, 0x1000, 0x1000, 0x4004549be0, 0x469b10, 0x4002470f00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010ca0, 0x400453b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40003b2fc0, 0x400453b000, 0x1000, 0x1000, 0x4004549cf8, 0x67867c, 0x40045129c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002469020) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4002469020, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40003b2fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 48 [select]: net/http.(*persistConn).writeLoop(0x40003b2fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 31 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x40023d53e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 32 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x4002394080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 33 [IO wait]: internal/poll.runtime_pollWait(0xffff7d385f38, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4002394198, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4002394180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4002394180, 0x0, 0x4000111e98, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40044ce978, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40044ce978, 0x12ed9d0, 0x40044ce978, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x4002394100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 98 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x4004518d00, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 99 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x4004518d00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 100 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x400246eea0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 108 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001fdb80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 78 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 103 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40001fa850) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 77 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001943c0, 0x1144559, 0x1b, 0x4004575e38, 0x400461ed80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004c14770, 0x400461ed80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004575f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40033e7920, 0x4004c14770) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 76 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x4004712460, 0x400461ec60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 95 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40000e8d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 114 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40013ae440) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 184 [IO wait]: internal/poll.runtime_pollWait(0xffff7c7cca70, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4003462d18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4003462d00, 0x4004cd4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4003462d00, 0x4004cd4000, 0x1000, 0x1000, 0x4004571be8, 0x7ff1a8, 0x4004571cb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010840, 0x4004cd4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400342aea0, 0x4004cd4000, 0x1000, 0x1000, 0x60, 0x0, 0x4004c90301) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004c94780) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004c94780, 0x1, 0x4004b55b60, 0x43d584, 0x0, 0xd8c91c, 0x4004571d68) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400342aea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 109 [IO wait]: internal/poll.runtime_pollWait(0xffff7d385d68, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4003462118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4003462100, 0x4004456000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4003462100, 0x4004456000, 0x1000, 0x1000, 0x4004544a48, 0x652ce0, 0x12edac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400237e068, 0x4004456000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400342a480, 0x4004456000, 0x1000, 0x1000, 0x113674e, 0x412bc0, 0x400008ce10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40033e6660) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40033e6660, 0x100a, 0x0, 0x0, 0x4000111b98, 0x4af370, 0x4000111c88) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40033e6660, 0x7356e4, 0xffffcc94094e, 0xe, 0x4000111ca8, 0x40b2f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004c92540) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004c92540, 0x4004bd4e00, 0x200, 0x200, 0x60, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40013c31c0, 0x4004bd4e00, 0x200, 0x200, 0x4002397e00, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40013c31c0, 0x4004bd4e00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40013c3240, 0x4004bd4e00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004c8c6e0, 0x680d00, 0x4000111eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004c8c6e0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004c8c6e0, 0xee0d40, 0x400008cd20, 0x12db068, 0x40013c3240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004c8a3f0, 0x4004c94180, 0x4004b551a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 117 [IO wait]: internal/poll.runtime_pollWait(0xffff7d385b98, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4002395418, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4002395400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4002395400, 0x12d9d48, 0x40013c25c0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004709248, 0x0, 0x40045128a0, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004709248, 0x400245db00, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x4004736100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 118 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x4004736540, 0x400007f1f0, 0x400007f260, 0x4004758130, 0x12e8f90, 0x40043ec7e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x4004c867e4, 0x5, 0x4004736100, 0x12eca70, 0x40013c3180, 0x130d478, 0x40000e0070, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40003e47e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40003e47e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 119 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40043ec7e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 120 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40043ec7e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 121 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x4003401c20, 0x40047341e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 122 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x4003412a80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 123 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x4004712460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 124 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x4004712460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 125 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x40013a4518) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 126 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x4004712500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 127 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x4004712500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 128 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x4004736540, 0x4004758130) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 80 [IO wait]: internal/poll.runtime_pollWait(0xffff7d385ab0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40033ccc98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40033ccc80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40033ccc80, 0x0, 0x4002368d88, 0x665fe4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c12888, 0x611a6d70, 0x4002368db8, 0x445c24) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c12888, 0x4002368e20, 0x4002368e28, 0x18, 0x4004c24000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40013ec700, 0x12ed9d0, 0x4004c12888, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40013ec700, 0x12ed9d0, 0x4004c12888, 0x40003a0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 81 [IO wait]: internal/poll.runtime_pollWait(0xffff7d3859c8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40033ccd98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40033ccd80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40033ccd80, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c128a0, 0xffffa7a30a68, 0xffff7c40b668, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c128a0, 0x665e1c, 0x0, 0xfe5e60, 0x400456e5c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004c128b8, 0x4004c166c0, 0x11ae500, 0x40013ec860, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40013ec7e0, 0x12ec710, 0x4004c128b8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40013ec7e0, 0x12ec710, 0x4004c128b8, 0x40003a0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 94 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40000e8d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 179 [select]: net/http.(*persistConn).writeLoop(0x40046c4240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 111 [IO wait]: internal/poll.runtime_pollWait(0xffff7c7ccd28, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4003462b18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4003462b00, 0x4004cac000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4003462b00, 0x4004cac000, 0x1000, 0x1000, 0x4004543be0, 0x469b10, 0x400475ad80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010720, 0x4004cac000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400031bb00, 0x4004cac000, 0x1000, 0x1000, 0x4004543cf8, 0x67867c, 0x4004b55380) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004c941e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004c941e0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400031bb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 181 [select]: net/http.(*persistConn).roundTrip(0x400342aea0, 0x40013c3500, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004bc4b00, 0xf8, 0x10a8660, 0x4004db9798) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004bc4b00, 0x1b6e900, 0xc03eba68270b3504, 0x117701ff01b) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004bc4a00, 0x12db048, 0x1b6e900, 0xc03eba68270b3504, 0x117701ff01b, 0x1be4a80, 0x0, 0x4000010818, 0x4004c929c0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004c92990, 0x4004bc4a00, 0xc03eba68270b3504, 0x117701ff01b, 0x1be4a80, 0x4000010818, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004c92990, 0x4004bc4a00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004c90300, 0x1f, 0x4004db9d10, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/indexer.getWithAuth(0x4004c90300, 0x1f, 0xe, 0x113846e, 0x11) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:3077 +0x68 github.com/couchbase/indexing/secondary/indexer.getGlobalTopology(0x4004dd5170, 0xe, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:2266 +0x78 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).updateNodeList(0x400471d800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:220 +0x3c created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:206 +0x98 goroutine 112 [select]: net/http.(*persistConn).writeLoop(0x400031bb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 93 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40000e8d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 92 [semacquire]: sync.runtime_Semacquire(0x4004dd5b58) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004dd5b50) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x40043e9b80, 0x40013a2600, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffcc940969, 0x29, 0xffffcc9409cb, 0x25, 0x4004735f80, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x400472b2e0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffcc940969, 0x29, 0xffffcc9409cb, 0x25, 0x4004735f80, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x400472b2e0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004ec3190, 0x4004ec2e60, 0x40013a2600, 0x400036ad80, 0x40013c3200, 0x0, 0xffff7c421508, 0x98, 0xffffa7a305e0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x40003a0000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x400472b2e0, 0xb, 0x400472b2d8, 0x6, 0x400472b2f0, 0xd, 0x4004707280, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x40003a0000, 0x4000320000, 0x400011a060) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x40003a0000, 0x40003b0640, 0x2, 0x2, 0x40003b0650, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 136 [select]: net/http.(*persistConn).writeLoop(0x40046c4120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 96 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40000e8d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 97 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4003413580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 152 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40000c5810, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 153 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40000c5810, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 154 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x40000c5810) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 137 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).processSchedIndexes(0x4003407c30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:296 +0x44 created by github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:218 +0x3c goroutine 155 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40000e8d20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 156 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40001943c0, 0x1144559, 0x1b, 0x40013f5e38, 0x4004e00180, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004758920, 0x4004e00180, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40013f5f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40046d4180, 0x4004758920) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 135 [IO wait]: internal/poll.runtime_pollWait(0xffff7c7ccc40, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40046b4498, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40046b4480, 0x4004c64000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40046b4480, 0x4004c64000, 0x1000, 0x1000, 0x4004d77be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400237e1e8, 0x4004c64000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40046c4120, 0x4004c64000, 0x1000, 0x1000, 0x4004d77cf8, 0x67867c, 0x400461f620) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004c56360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004c56360, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40046c4120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 180 [IO wait]: internal/poll.runtime_pollWait(0xffff7c7cc8a0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40033cd218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40033cd200, 0x4004c69000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40033cd200, 0x4004c69000, 0x1000, 0x1000, 0x4004723848, 0x453294, 0xf1f900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400237e210, 0x4004c69000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x4004c568a0, 0x4004605e00, 0x200, 0x200, 0x200, 0xf1f980, 0x4004723801) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x4004c129a8, 0x4004605e00, 0x200, 0x200, 0x8, 0x400237e2d8, 0x400005a800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x4004c1ba20, 0xfe22a0, 0x4004723948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004c1ba20, 0x0, 0x0, 0x412bc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004c1ba20, 0xf5a960, 0x4004c56d40, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4004c56d20, 0x400237cbc0, 0x4004723a48, 0x486a6c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x40033cb040, 0x12f8f98, 0x4004c56d20, 0x6eefc8, 0xf685c0, 0x4004c17a10, 0x4004723c08, 0x412bc0, 0xf7ec00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x40033cb040, 0x12f8f98, 0x4004c56d20, 0x412ff8, 0x4004c17a40, 0x30, 0x30, 0x1063520, 0x4004723ce8, 0x412ff8, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x40033cb040, 0x12f8f98, 0x4004c56d20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4004c17650, 0x4004723f10, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x4004723f10, 0x4004c17650, 0x0, 0x0, 0x0, 0xd8d03c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth/service.RegisterManager(0x13007d8, 0x400471d800, 0x0, 0x0, 0x4004d79f80, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:146 +0xa0 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).registerWithServer(0x400471d800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:301 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:205 +0x78 goroutine 178 [select]: net/http.(*persistConn).readLoop(0x40046c4240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 166 [IO wait]: internal/poll.runtime_pollWait(0xffff7c7ccb58, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40046b4598, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40046b4580, 0x4004cc6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40046b4580, 0x4004cc6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000107f8, 0x4004cc6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40046c4240, 0x4004cc6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004c943c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004c943c0, 0x100a, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004c943c0, 0x0, 0x0, 0x0, 0x400475eca8, 0x40b2f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000371b60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000371b60, 0x4004681a00, 0x200, 0x200, 0x60, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4003413680, 0x4004681a00, 0x200, 0x200, 0x4004e00ae0, 0x43d584, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4003413680, 0x4004681a00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40034136c0, 0x4004681a00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40046fe840, 0x680d00, 0x400475eeb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40046fe840, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40046fe840, 0xee0d40, 0x4003434aa0, 0x12db068, 0x40034136c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004e0c120, 0x40046d49c0, 0x4004e00b40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 157 [select]: net/http.(*persistConn).roundTrip(0x40003b2fc0, 0x40013af140, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004c18a00, 0xf8, 0x10a8660, 0x4004c44e18) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004c18a00, 0x1b6e900, 0xc03eb95a270bca4f, 0x1bfb1c555e) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004c18900, 0x12db048, 0x1b6e900, 0xc03eb95a270bca4f, 0x1bfb1c555e, 0x1be4a80, 0x0, 0x400237e208, 0x4004c17560, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004c17530, 0x4004c18900, 0xc03eb95a270bca4f, 0x1bfb1c555e, 0x1be4a80, 0x400237e208, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004c17530, 0x4004c18900, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x400236be30, 0x1b, 0x4004c45580, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004c265a0, 0x112f905, 0x6, 0x12db908, 0x400237cb60, 0xee40c0, 0x40033cd118, 0x4004c45580, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x4003436280, 0x46, 0x12db908, 0x400237cb60, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x4003436280, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x20, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004c45d40, 0x4, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x40046ba6c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x40046ba6c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40034135c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1267 +0xf8 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 158 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x4003407c30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:224 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 159 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x4003407c30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:612 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 160 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x4003407c30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:662 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 161 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x4003407c30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:752 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 194 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x400471d800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:249 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 186 [select]: net/http.(*persistConn).roundTrip(0x400342afc0, 0x40013c3d00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004bc4e00, 0xf8, 0x10a8660, 0x4004576588) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004bc4e00, 0x1b6e900, 0xc03eb95a270e8cae, 0x1bfb1f17c5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004bc4d00, 0x12db048, 0x1b6e900, 0xc03eb95a270e8cae, 0x1bfb1f17c5, 0x1be4a80, 0x0, 0x4000010878, 0x4004c92ea0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004c92e70, 0x4004bc4d00, 0xc03eb95a270e8cae, 0x1bfb1f17c5, 0x1be4a80, 0x4000010878, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004c92e70, 0x4004bc4d00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x400453f0e0, 0x1b, 0x4004576cf0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004c8aa20, 0x112f905, 0x6, 0x12db908, 0x4000046f20, 0xee40c0, 0x4003462e98, 0x4004576cf0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x4004c96200, 0x46, 0x12db908, 0x4000046f20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x4004c96200, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x4004c92db0, 0xffff7c6bbf18) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004b614b0, 0x1063520, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x400457e480, 0x400457e480, 0x4004c96200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.FetchNewClusterInfoCache(0xffffcc94094e, 0xe, 0x113075a, 0x7, 0x1130e44, 0x8, 0x4004cd60c4, 0x2, 0xf01) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:123 +0x134 github.com/couchbase/indexing/secondary/manager.(*IndexManager).FetchNewClusterInfoCache(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:306 github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).getIndexMetadata(0x1be5520, 0x12ec7a0, 0x40013c3cc0, 0x400008d360, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:1284 +0x74 github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).handleIndexMetadataRequest(0x1be5520, 0x12edbe0, 0x40003e48c0, 0x4004bc4c00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:1271 +0x3e8 net/http.HandlerFunc.ServeHTTP(0x4004c14670, 0x12edbe0, 0x40003e48c0, 0x4004bc4c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2069 +0x40 net/http.(*ServeMux).ServeHTTP(0x40013aed80, 0x12edbe0, 0x40003e48c0, 0x4004bc4c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2448 +0x190 net/http.serverHandler.ServeHTTP(0x40013ec700, 0x12edbe0, 0x40003e48c0, 0x4004bc4c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2887 +0xbc net/http.(*conn).serve(0x40013caaa0, 0x12f86a0, 0x40013c3580) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:1952 +0x71c created by net/http.(*Server).Serve /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:3013 +0x318 goroutine 196 [select]: net/http.(*persistConn).roundTrip(0x40046c4120, 0x40013af100, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004c18800, 0xf8, 0x10a8660, 0x4004724dd8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004c18800, 0x1b6e900, 0xc03eb95a270ab0bd, 0x1bfb1b3bd4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004c18700, 0x12db048, 0x1b6e900, 0xc03eb95a270ab0bd, 0x1bfb1b3bd4, 0x1be4a80, 0x0, 0x400237e1f8, 0x4004c173e0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004c173b0, 0x4004c18700, 0xc03eb95a270ab0bd, 0x1bfb1b3bd4, 0x1be4a80, 0x400237e1f8, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004c173b0, 0x4004c18700, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x400236bd10, 0x1b, 0x4004725540, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004c262d0, 0x112f905, 0x6, 0x12db908, 0x400237cb20, 0xee40c0, 0x40033cd098, 0x4004725540, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x40023d2200, 0x46, 0x12db908, 0x400237cb20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x40023d2200, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x240, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004725d00, 0x4004c172f0, 0xffff7c7b9bb8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x40013b5440, 0xd47600, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x40013b5440, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x40003a0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9702 +0x344 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 197 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x40003a0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1090 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 198 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x40003a0000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1017 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 199 [sleep]: time.Sleep(0x12a05f200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/plasma.RunMemQuotaTunerIteration(0x4004d79fb0, 0xdf74d8, 0x40013aa5a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:211 +0x24c github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:247 +0xc0 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 goroutine 185 [select]: net/http.(*persistConn).writeLoop(0x400342aea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 187 [IO wait]: internal/poll.runtime_pollWait(0xffff7c7cc988, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4003462d98, 0x72, 0x0, 0x1, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4003462d80, 0x4004c92c41, 0x1, 0x1, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4003462d80, 0x4004c92c41, 0x1, 0x1, 0x1, 0x0, 0x4004b55260) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010850, 0x4004c92c41, 0x1, 0x1, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*connReader).backgroundRead(0x4004c92c30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:692 +0x50 created by net/http.(*connReader).startBackgroundRead /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:688 +0xc4 goroutine 190 [IO wait]: internal/poll.runtime_pollWait(0xffff7c7cc7b8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4003462f98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4003462f80, 0x4004cda000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4003462f80, 0x4004cda000, 0x1000, 0x1000, 0x4004571be8, 0x7ff1a8, 0x4004571cb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010898, 0x4004cda000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400342afc0, 0x4004cda000, 0x1000, 0x1000, 0x60, 0x0, 0x4004c90301) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004c94d20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004c94d20, 0x1, 0x4004b55f80, 0x43d584, 0x0, 0xd8c91c, 0x4004571d68) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400342afc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 191 [select]: net/http.(*persistConn).writeLoop(0x400342afc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc Initializing write barrier = 8000 2021-08-16T13:51:49.829+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:51:49.829+00:00 [Info] Setting ipv6=false 2021-08-16T13:51:49.834+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:51:49.838+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:51:49.838+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:51:49.838+00:00 [Info] Setting log level to Info 2021-08-16T13:51:49.840+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021-08-16T13:51:49.841+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021/08/16 13:51:49 audit: created new audit service 2021-08-16T13:51:49.850+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:51:49.850+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:51:49.851+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:51:49.851+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:51:49.851+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:51:49.862+00:00 [Info] security context initialized 2021-08-16T13:51:49.862+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:51:49.863+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:51:49.863+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:51:49.863+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:51:49.863+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:51:49.879+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:51:49.879+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:51:49.881+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:51:49.904+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:51:49.904+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:51:49.905+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:51:49.906+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:51:49.907+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:51:49.908+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:51:49.908+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:51:49.910+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:51:49.959+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:51:49.959+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 68 us 2021-08-16T13:51:49.959+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:51:49.961+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:51:49.962+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:51:49.962+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:51:49.962+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:51:49.962+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:51:49.962+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:51:49.962+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:51:49.968+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:49.968+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:49.970+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:51:49.970+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:51:49.970+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:51:49.970+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:49.970+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:51:49.970+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:51:49.970+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:51:49.970+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:51:49.970+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:51:49.970+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:51:49.970+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:51:49.974+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 3.811779ms 2021-08-16T13:51:49.974+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:51:49.975+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:51:49.975+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:49.975+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:51:49.976+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:51:49.976+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:51:49.976+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:51:49.987+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:51:49.987+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:51:49.987+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:51:49.987+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:51:49.987+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x400046945f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x400046945f pc=0xbd8ea4] goroutine 183 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004ec29f0 sp=0x4004ec29c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004ec2a30 sp=0x4004ec29f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x40001dd6c0, 0x4004ed6000, 0x11a9890, 0x4004ec2b40, 0x40001dd6d8, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004ec2ab0 sp=0x4004ec2a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x40001dd6c0, 0x4004ed6000, 0x11a9890, 0x4004ec2b40, 0x40001dd6d8, 0xc18f8c, 0x4004ec2c18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004ec2af0 sp=0x4004ec2ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4000032dc0, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004ec2da0 sp=0x4004ec2af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4000032dc0, 0x4000126e00, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004ec2e60 sp=0x4004ec2da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x40045e6b80, 0x40045e6b90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004ec3200 sp=0x4004ec2e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x40045e6b80, 0x40045e6b90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004ec38f0 sp=0x4004ec3200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x40045f9a20, 0x40045dbc00, 0x400249d080, 0x40045e6b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004ec3fb0 sp=0x4004ec38f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004ec3fb0 sp=0x4004ec3fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x4000232000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x40000feed0, 0x4, 0x1136a87, 0xf, 0x400019de68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 7 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400000e678) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 8 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400000e690) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 9 [IO wait]: internal/poll.runtime_pollWait(0xffff6c131e88, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000216098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000216080, 0x40001c3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000216080, 0x40001c3000, 0x1000, 0x1000, 0x1311860, 0x47b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001a0028, 0x40001c3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x40001942a0, 0x40001d4001, 0xdff, 0xdff, 0x400009b8e8, 0x6902d8, 0x4000199948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400019e030, 0x40001d4001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40001c0160, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40001c0160, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40001c0160, 0xf5a960, 0x4000194380, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4000194360, 0x40001a41a0, 0x4000199b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x40001ce000, 0x12f8f98, 0x4000194360, 0x40001a28e7, 0x16, 0x40001b6480, 0x400009bc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x40001ce000, 0x12f8f98, 0x4000194360, 0x40001ce000, 0x40001a20f0, 0x40001a2100, 0x40001aa080, 0x40001a40c0, 0xee0980, 0x40001b6480, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x40001ce000, 0x12f8f98, 0x4000194360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4000097e30, 0x400009bf78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x40004a7f78, 0x4000097e30, 0x12db8a8, 0x400020a000, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x4000097e30, 0x40000b12c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x4000097e30, 0x40000b12c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 19 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400018c200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 10 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 11 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 12 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 13 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 14 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 15 [syscall]: syscall.Syscall(0x3f, 0x0, 0x400044cfa0, 0x4, 0x400044ce68, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x400044cfa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x400044cfa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x400044cfa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 56 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40002942a0, 0x12c18bc, 0x1, 0x40004a9e58, 0x40002306c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x4000052ec8, 0x40002306c0, 0x4000052ec8, 0x679704) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40004a9fa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x400023a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 54 [select]: net/http.(*persistConn).readLoop(0x40001b6120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 55 [select]: net/http.(*persistConn).writeLoop(0x40001b6120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 57 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x400023a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 66 [IO wait]: internal/poll.runtime_pollWait(0xffff6c131da0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000216198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000216180, 0x40001d5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000216180, 0x40001d5000, 0x1000, 0x1000, 0x40004afa88, 0x68f7a4, 0x400012a050) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000206030, 0x40001d5000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001b6120, 0x40001d5000, 0x1000, 0x1000, 0xf12180, 0x400008c090, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000200420) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000200420, 0xa, 0x1008880, 0x400008c050, 0x199, 0x1000, 0x40004afb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000200420, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40000fe0c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40000fe0c0, 0x4000130000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000456140, 0x4000130000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000456140, 0x4000130000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000456200, 0x4000130000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x400012a000, 0x0, 0x40004afeb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400012a000, 0x0, 0x0, 0x40004aff70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400012a000, 0xee0d40, 0x400008c050, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x400011a000, 0x40001946c0, 0x4000128000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 60 [IO wait]: internal/poll.runtime_pollWait(0xffff6c131cb8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000216298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000216280, 0x4000276000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000216280, 0x4000276000, 0x1000, 0x1000, 0x400006abe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000206078, 0x4000276000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001b6900, 0x4000276000, 0x1000, 0x1000, 0x400006acf8, 0x67867c, 0x4000230de0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000200840) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000200840, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40001b6900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 61 [select]: net/http.(*persistConn).writeLoop(0x40001b6900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 39 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400023a640) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 37 [select]: net/http.(*persistConn).readLoop(0x400000cfc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 38 [select]: net/http.(*persistConn).writeLoop(0x400000cfc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 152 [semacquire]: sync.runtime_Semacquire(0x40045f9a28) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x40045f9a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x400249d080, 0x4000202a50, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xfffff2c7e969, 0x29, 0xfffff2c7e9cb, 0x25, 0x40045dcfc0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004755ac0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xfffff2c7e969, 0x29, 0xfffff2c7e9cb, 0x25, 0x40045dcfc0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004755ac0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004ec3190, 0x4004ec2e60, 0x4000202a50, 0x40001b8fc0, 0x400023a640, 0x0, 0x17f, 0x4004d82da8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x4000232000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004755ac0, 0xb, 0x4004755ab8, 0x6, 0x4004755ad0, 0xd, 0x400476adc0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x4000232000, 0x4000420000, 0x40002020f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x4000232000, 0x40044b6440, 0x2, 0x2, 0x40044b6450, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 92 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x400467e640) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 93 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x400467e640) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 41 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x40000e87e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 20 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x40000e87e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 21 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x40000e87e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 22 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x40000e87e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 94 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x40001a9f00, 0x4004bae970) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 44 [select]: net/http.(*persistConn).readLoop(0x40024ba7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 45 [select]: net/http.(*persistConn).writeLoop(0x40024ba7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 23 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001a9280) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 24 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x400010e6c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 25 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x400022a4b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 26 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x40001aa100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 27 [IO wait]: internal/poll.runtime_pollWait(0xffff6c131660, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001aa318, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40001aa300, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40001aa300, 0x0, 0x4000054698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40044b1a40, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40044b1a40, 0x12ed9d0, 0x40044b1a40, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x40001aa200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 28 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40000e4c30, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 29 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40000e4c30, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 30 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x40000e4c30) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 31 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x400022b9b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 48 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x4000280000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 153 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40001c6e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 82 [IO wait]: internal/poll.runtime_pollWait(0xffff6c1313a8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001aa698, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40001aa680, 0x4000131000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40001aa680, 0x4000131000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001a0220, 0x4000131000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40043bc5a0, 0x4000131000, 0x1000, 0x1000, 0x4000102600, 0x200000003, 0x4000102600) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400010f380) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400010f380, 0x40004bc00a, 0x40045658c8, 0x4d3b10, 0x40000820c0, 0x40004bc000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400010f380, 0x0, 0x1, 0x0, 0x40000820d8, 0x40004bc000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40001f1230) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40001f1230, 0x400015e000, 0x1000, 0x1000, 0x40045659b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400023a2c0, 0x400015e000, 0x1000, 0x1000, 0x400010eb98, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400023a2c0, 0x400015e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400023a300, 0x400015e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4004565dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004565dd0, 0xa, 0x400015e201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4004565dd0, 0x400015e20a, 0x1, 0x400011c919, 0x0, 0x400015e2e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4004565dd0, 0x400009840a, 0x400011c919, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000098380, 0x400003a9f0, 0x17, 0x4004565f20, 0x40044b1ab8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x4000098380, 0x113075a, 0x7, 0x40044b1ab8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x4000098340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 83 [IO wait]: internal/poll.runtime_pollWait(0xffff6c131578, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001ab318, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40001ab300, 0x40024fe000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40001ab300, 0x40024fe000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001a05e0, 0x40024fe000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40024ba7e0, 0x40024fe000, 0x1000, 0x1000, 0x4000102780, 0x200000003, 0x4000102780) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400010fc80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400010fc80, 0x40004bc00a, 0x400009f8c8, 0x4d3b10, 0x40000820c0, 0x40004bc000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400010fc80, 0x0, 0x1, 0x0, 0x40000820d8, 0x40004bc000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400022be90) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400022be90, 0x4004567000, 0x1000, 0x1000, 0x400009f9b8, 0x0, 0x400009f988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40001a84c0, 0x4004567000, 0x1000, 0x1000, 0x4000182480, 0x3, 0x400009fbe0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40001a84c0, 0x4004567000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40001a8500, 0x4004567000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400009fdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400009fdd0, 0xa, 0x4004567201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400009fdd0, 0x400456720a, 0x1, 0x40001eb82b, 0x0, 0x4004567263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400009fdd0, 0x400004650a, 0x40001eb82b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000098380, 0x40000425a0, 0x24, 0x400009ff20, 0x40044b1aa0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x4000098380, 0x113075a, 0x7, 0x40044b1aa0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x4000098340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 86 [select]: net/http.(*persistConn).readLoop(0x40043bc5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 87 [select]: net/http.(*persistConn).writeLoop(0x40043bc5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 71 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40044540c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 137 [IO wait]: internal/poll.runtime_pollWait(0xffff6c131490, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004df8298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004df8280, 0x4004e26000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004df8280, 0x4004e26000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001a06f8, 0x4004e26000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40043bdb00, 0x4004e26000, 0x1000, 0x1000, 0x5, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004db91a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004db91a0, 0xa, 0x4000110123, 0x11320c7, 0x9, 0x1, 0x400445d475) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004db91a0, 0xd4fa28, 0x400003e400, 0x46, 0x11320c7, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400477b0b0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400477b0b0, 0x4004ca0400, 0x200, 0x200, 0x400477e830, 0x400477e830, 0x4000122870) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004c8c4c0, 0x4004ca0400, 0x200, 0x200, 0x4004674107, 0x33, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004c8c4c0, 0x4004ca0400, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004c8c500, 0x4004ca0400, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004c9edc0, 0x0, 0x400445d6b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004c9edc0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004c9edc0, 0xee0d40, 0x40043c0eb0, 0x12db068, 0x4004c8c500) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004745830, 0x4004cb0000, 0x40047195c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 49 [IO wait]: internal/poll.runtime_pollWait(0xffff6c1311d8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000216418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000216400, 0x40044bb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000216400, 0x40044bb000, 0x1000, 0x1000, 0x400009d688, 0x69f178, 0x400009d6f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000104e8, 0x40044bb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40004105a0, 0x40044bb000, 0x1000, 0x1000, 0x5, 0x4000482000, 0x400009d7c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000195260) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000195260, 0x68f90a, 0x40001da4d0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000195260, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000202990) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000202990, 0x400446e000, 0x1000, 0x1000, 0x400009d998, 0x44e884, 0x400009d948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004454200, 0x400446e000, 0x1000, 0x1000, 0x400010eb98, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004454200, 0x400446e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004454240, 0x400446e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400009dd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400009dd80, 0xa, 0x400446e201, 0x1, 0xffff6c1b0308, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400009dd80, 0x400446e20a, 0x1, 0x4002412650, 0x0, 0x400446e259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400009dd80, 0x40001c090a, 0x4002412650, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000098380, 0x4000044a40, 0x1f, 0x400009dee0, 0x400019e7e0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x4000098380, 0x113075a, 0x7, 0x400011c930, 0xd, 0x400019e7e0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x4000098340, 0x400011c930, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 102 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40002942a0, 0x1144559, 0x1b, 0x40004a5e38, 0x40045dc4e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x40045e6180, 0x40045dc4e0, 0x40043bc480, 0x4000448640) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x400240c6f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40004a5f80, 0x12c2e00, 0x1000000000001) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40045e0180, 0x40045e6180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 100 [select]: net/http.(*persistConn).readLoop(0x40004105a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 101 [select]: net/http.(*persistConn).writeLoop(0x40004105a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 72 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400023a540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 73 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400023b4c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 74 [IO wait]: internal/poll.runtime_pollWait(0xffff6c1310f0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1c18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f1c00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f1c00, 0x58ec48, 0x40024fc060, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40047693f8, 0x4714f4, 0x40024fc060, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40047693f8, 0x40024b5900, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x400023bd40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 75 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x40001a9f00, 0x40001faee0, 0x40001faf50, 0x4004bae970, 0x12e8f90, 0x40043fe7e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x4004754dd4, 0x5, 0x400023bd40, 0x12eca70, 0x400023bc00, 0x130d478, 0x40000e0070, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40043fea80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40043fea80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 76 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40043fe7e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 77 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40043fe7e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 78 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x400243a120, 0x4004718ae0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 79 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x4004454080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 80 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x40043c52c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 81 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x40043c52c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 130 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x40043a2478) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 131 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x40043c52c0, 0x4004718b40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 103 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 104 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400008caf0, 0xd00007, 0x40024b53f8, 0x5, 0x40045e6190, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 105 [IO wait]: internal/poll.runtime_pollWait(0xffff6c1312c0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000216818, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000216800, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000216800, 0xffff94a68108, 0x30, 0x40045a8a50) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40045d8720, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40045d8720, 0x40045a8a50, 0x11ae500, 0x40044dc400, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40044dc380, 0x12ed9d0, 0x40045d8720, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40044dc380, 0x12ed9d0, 0x40045d8720, 0x4000232000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 106 [IO wait]: internal/poll.runtime_pollWait(0xffff6c131008, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000216a18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000216a00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000216a00, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x40045d8738, 0xffff94a68108, 0xffff6c0f1ca0, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x40045d8738, 0x665e1c, 0x0, 0xfe5e60, 0x40045f15c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x40045d8750, 0x40045a8ae0, 0x11ae500, 0x40044dc4e0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40044dc460, 0x12ec710, 0x40045d8750, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40044dc460, 0x12ec710, 0x40045d8750, 0x4000232000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 132 [IO wait]: internal/poll.runtime_pollWait(0xffff6c131748, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001aab98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40001aab80, 0x4000015000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40001aab80, 0x4000015000, 0x1000, 0x1000, 0x400455ba68, 0xbdff1c, 0xf6c340) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001a0330, 0x4000015000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000cfc0, 0x4000015000, 0x1000, 0x1000, 0x400477c4b0, 0x400477c3e0, 0x4000161958) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400010efc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400010efc0, 0x55b60a, 0xf1f980, 0x0, 0x37, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400010efc0, 0x0, 0x1, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400477a870) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400477a870, 0x40046f5c00, 0x200, 0x200, 0x2, 0xe, 0x400026d410) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004c8c100, 0x40046f5c00, 0x200, 0x200, 0xfffff2c7e9cb, 0x0, 0x4000202a50) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004c8c100, 0x40046f5c00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004c8c140, 0x40046f5c00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004717b80, 0x0, 0x400445beb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004717b80, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004717b80, 0xee0d40, 0x40043c0cd0, 0x12db068, 0x4004c8c140) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004745200, 0x4000161b00, 0x4004718ba0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 134 [IO wait]: internal/poll.runtime_pollWait(0xffff6c1ce0d8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f1e98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f1e80, 0x400476f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f1e80, 0x400476f000, 0x1000, 0x1000, 0x400455cbe0, 0x469b10, 0x4000102d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40043a24a0, 0x400476f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40043bd7a0, 0x400476f000, 0x1000, 0x1000, 0x400455ccf8, 0x67867c, 0x4004718d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000161b60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000161b60, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40043bd7a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 154 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40001c6e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 135 [select]: net/http.(*persistConn).writeLoop(0x40043bd7a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 150 [IO wait]: internal/poll.runtime_pollWait(0xffff6c1cdff0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004df8098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004df8080, 0x4004dcf000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004df8080, 0x4004dcf000, 0x1000, 0x1000, 0x4002435be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001a0628, 0x4004dcf000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40043bd8c0, 0x4004dcf000, 0x1000, 0x1000, 0x4002435cf8, 0x67867c, 0x4004dfc0c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004db83c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004db83c0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40043bd8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 151 [select]: net/http.(*persistConn).writeLoop(0x40043bd8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 108 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4004bfc580) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 155 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40001c6e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 156 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40001c6e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 157 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4004bfc540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 158 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004da8630, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 159 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004da8630, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 160 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4004da8630) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 161 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40001c6e00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 162 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x40002942a0, 0x1144559, 0x1b, 0x4004d85e38, 0x4004dfc660, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004baf2b0, 0x4004dfc660, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004d85f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004db8720, 0x4004baf2b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 165 [select]: net/http.(*persistConn).readLoop(0x40043bdb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 166 [select]: net/http.(*persistConn).writeLoop(0x40043bdb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 109 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x4004da84d0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 110 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x4004da84d0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 111 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x4004da84d0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 112 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x4004da84d0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 113 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x40045fa500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 178 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x40045fa500, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 179 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x4000232000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 180 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x4000232000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 181 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x4000232000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 182 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:51:55.250+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:51:55.250+00:00 [Info] Setting ipv6=false 2021-08-16T13:51:55.257+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:51:55.263+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:51:55.263+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:51:55.263+00:00 [Info] Setting log level to Info 2021-08-16T13:51:55.263+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:51:55.266+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:51:55 audit: created new audit service 2021-08-16T13:51:55.290+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:51:55.290+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:51:55.290+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:51:55.290+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:51:55.290+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:51:55.309+00:00 [Info] security context initialized 2021-08-16T13:51:55.309+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:51:55.309+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:51:55.309+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:51:55.309+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:51:55.309+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:51:55.337+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:51:55.337+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:51:55.345+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:51:55.368+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:51:55.370+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:51:55.370+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:51:55.371+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:51:55.372+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:51:55.373+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:51:55.373+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:51:55.376+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:51:55.437+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:51:55.437+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 65 us 2021-08-16T13:51:55.437+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:51:55.440+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:51:55.440+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:51:55.453+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:51:55.459+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:51:55.460+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:51:55.460+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:51:55.460+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:51:55.461+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:55.461+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:55.462+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:51:55.462+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:51:55.462+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:51:55.463+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:55.463+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:51:55.463+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:51:55.463+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:51:55.463+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:51:55.463+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:51:55.463+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:51:55.463+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:51:55.467+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 3.326711ms 2021-08-16T13:51:55.467+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:51:55.468+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:51:55.468+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:55.468+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:51:55.469+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:51:55.469+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:51:55.469+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:51:55.480+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:51:55.481+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:51:55.482+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:51:55.482+00:00 [Info] RebalanceMgr::initService Init 2021-08-16T13:51:55.482+00:00 [Info] ServiceMgr::registerWithServer nodeuuid 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:51:55.483+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:51:55.483+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x4002507d5f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x4002507d5f pc=0xbd8ea4] goroutine 158 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004ca09f0 sp=0x4004ca09c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004ca0a30 sp=0x4004ca09f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x40000cd6c0, 0x400471ac00, 0x11a9890, 0x4004ca0b40, 0x40000cd6d8, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004ca0ab0 sp=0x4004ca0a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x40000cd6c0, 0x400471ac00, 0x11a9890, 0x4004ca0b40, 0x40000cd6d8, 0x28, 0x4004ca0c18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004ca0af0 sp=0x4004ca0ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4000144b40, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004ca0da0 sp=0x4004ca0af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4000144b40, 0x400015b340, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004ca0e60 sp=0x4004ca0da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004701270, 0x4004701280, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004ca1200 sp=0x4004ca0e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004701270, 0x4004701280, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004ca18f0 sp=0x4004ca1200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004e58960, 0x4004e80380, 0x4004455b80, 0x4004701250) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004ca1fb0 sp=0x4004ca18f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004ca1fb0 sp=0x4004ca1fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x400049a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x4000119a10, 0x4, 0x1136a87, 0xf, 0x4000125e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 34 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x4000126120) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 35 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x4000126138) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 36 [IO wait]: internal/poll.runtime_pollWait(0xffff8d473758, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000208118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000208100, 0x4000219000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000208100, 0x4000219000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b84e8, 0x4000219000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x40001ba360, 0x400021e001, 0xdff, 0xdff, 0x400019f8e8, 0x6902d8, 0x4000121948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x40001aa588, 0x400021e001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40001e66e0, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40001e66e0, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40001e66e0, 0xf5a960, 0x40001ba800, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x40001ba7e0, 0x40001c22e0, 0x4000121b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x4000180370, 0x12f8f98, 0x40001ba7e0, 0x40001b4c77, 0x16, 0x40001c7320, 0x400019fc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x4000180370, 0x12f8f98, 0x40001ba7e0, 0x4000180370, 0x40001b4488, 0x40001b4490, 0x4000208180, 0x40001c2200, 0xee0980, 0x40001c7320, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x4000180370, 0x12f8f98, 0x40001ba7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4000118990, 0x400019ff78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400019bf78, 0x4000118990, 0x12db8a8, 0x40001aa558, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x4000118990, 0x4000142460, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x4000118990, 0x4000142460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 37 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 38 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 39 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 40 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 41 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 20 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x4000190400) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 25 [select]: net/http.(*persistConn).writeLoop(0x40001c6360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 42 [syscall]: syscall.Syscall(0x3f, 0x0, 0x4000163fa0, 0x4, 0x4000163e68, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x4000163fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x40001ba000, 0x4000163fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x40001b8000, 0x4000163fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 55 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400023a800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 49 [select]: net/http.(*persistConn).readLoop(0x400000c120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 50 [select]: net/http.(*persistConn).writeLoop(0x400000c120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 5 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400011c800, 0x12c18bc, 0x1, 0x400019de58, 0x4000154b40, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x40000566c8, 0x4000154b40, 0xffffb7b86968, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x4000000101) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400019dfa0, 0x400007e000, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4000131300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 6 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4000131300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 11 [IO wait]: internal/poll.runtime_pollWait(0xffff8d473670, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000208098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000208080, 0x400021f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000208080, 0x400021f000, 0x1000, 0x1000, 0x400006aa88, 0x68f7a4, 0x400013e9f0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010008, 0x400021f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c120, 0x400021f000, 0x1000, 0x1000, 0xf12180, 0x400013a270, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40000821e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000821e0, 0xa, 0x1008880, 0x400013a230, 0x199, 0x1000, 0x400006ab00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40000821e0, 0x2, 0x0, 0x9, 0xaa6, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400044d560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400044d560, 0x40000b0000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000130100, 0x40000b0000, 0xe00, 0xe00, 0x36b, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000130100, 0x40000b0000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000130200, 0x40000b0000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x400013e9a0, 0x0, 0x400006aeb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400013e9a0, 0x0, 0x0, 0x400006af70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400013e9a0, 0xee0d40, 0x400013a230, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40001f6480, 0x40001bad20, 0x4000182480) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 9 [IO wait]: internal/poll.runtime_pollWait(0xffff8d473588, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000132118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000132100, 0x40000a4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000132100, 0x40000a4000, 0x1000, 0x1000, 0x40000acbe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b8310, 0x40000a4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c66c0, 0x40000a4000, 0x1000, 0x1000, 0x40000accf8, 0x67867c, 0x4000182420) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001baba0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40001baba0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40001c66c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 10 [select]: net/http.(*persistConn).writeLoop(0x40001c66c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 53 [select]: net/http.(*persistConn).readLoop(0x400000c480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 54 [select]: net/http.(*persistConn).writeLoop(0x400000c480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 15 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400249a5c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 74 [IO wait]: internal/poll.runtime_pollWait(0xffff8d472aa8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40014c2898, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40014c2880, 0x40004d4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40014c2880, 0x40004d4000, 0x1000, 0x1000, 0x40000a3688, 0x69f178, 0x40000a36f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40014ac118, 0x40004d4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400028ed80, 0x40004d4000, 0x1000, 0x1000, 0x5, 0x40014f4a00, 0x40000a37c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000083800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000083800, 0x68f90a, 0x40014bafd0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000083800, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400149f890) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400149f890, 0x40004da000, 0x1000, 0x1000, 0x40000a3998, 0x44e884, 0x40000a3948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40014b6b00, 0x40004da000, 0x1000, 0x1000, 0x40001ba9b8, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40014b6b00, 0x40004da000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40014b6b40, 0x40004da000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40000a3d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000a3d80, 0xa, 0x40004da201, 0x1, 0xffff8d48d8c0, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40000a3d80, 0x40004da20a, 0x1, 0x400455e320, 0x0, 0x40004da259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40000a3d80, 0x40004d850a, 0x400455e320, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000138380, 0x400049fc60, 0x1f, 0x40000a3ee0, 0x40004aa168, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x4000138380, 0x113075a, 0x7, 0x40004907b0, 0xd, 0x40004aa168, 0x0, 0x0, 0x2) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x4000138340, 0x40004907b0, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 98 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x400249d2c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 16 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x400041db00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 57 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x40002007e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 62 [IO wait]: internal/poll.runtime_pollWait(0xffff8d472f30, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000266118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000266100, 0x40014ca000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000266100, 0x40014ca000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001280a8, 0x40014ca000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000460120, 0x40014ca000, 0x1000, 0x1000, 0x4000001e00, 0x200000003, 0x4000001e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40001bb860) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001bb860, 0x40000b600a, 0x40001998c8, 0x4d3b10, 0x40001ba060, 0x40000b6000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40001bb860, 0x0, 0x1, 0x0, 0x40001ba078, 0x40000b6000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40002644e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40002644e0, 0x40014cc000, 0x1000, 0x1000, 0x40001999b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400023a2c0, 0x40014cc000, 0x1000, 0x1000, 0x4000083438, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400023a2c0, 0x40014cc000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400023a300, 0x40014cc000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000199dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000199dd0, 0xa, 0x40014cc201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000199dd0, 0x40014cc20a, 0x1, 0x4000490799, 0x0, 0x40014cc2e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000199dd0, 0x40014d120a, 0x4000490799, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000138380, 0x40001347f8, 0x17, 0x4000199f20, 0x40024fa1e0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x4000138380, 0x113075a, 0x7, 0x40024fa1e0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x4000138340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 12 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x40002007e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 13 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x40002007e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 14 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x40002007e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 85 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x4000252b40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 84 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x4000252b40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 63 [IO wait]: internal/poll.runtime_pollWait(0xffff8d473018, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000208d18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000208d00, 0x4000015000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000208d00, 0x4000015000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40000100c0, 0x4000015000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c480, 0x4000015000, 0x1000, 0x1000, 0x40024e6300, 0x200000003, 0x40024e6300) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000082fc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000082fc0, 0x40000b600a, 0x40000a18c8, 0x4d3b10, 0x40001ba060, 0x40000b6000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000082fc0, 0x0, 0x1, 0x0, 0x40001ba078, 0x40000b6000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400044d2c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400044d2c0, 0x4002460000, 0x1000, 0x1000, 0x40000a19b8, 0x0, 0x40000a1988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400249a900, 0x4002460000, 0x1000, 0x1000, 0x4000103c80, 0x3, 0x40000a1be0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400249a900, 0x4002460000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400249a940, 0x4002460000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40000a1dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000a1dd0, 0xa, 0x4002460201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40000a1dd0, 0x400246020a, 0x1, 0x40000362db, 0x0, 0x4002460263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40000a1dd0, 0x400012c70a, 0x40000362db, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000138380, 0x400011a2d0, 0x24, 0x40000a1f20, 0x40024fa1c8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x4000138380, 0x113075a, 0x7, 0x40024fa1c8, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x4000138340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 86 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x400023b640, 0x40047002f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 82 [select]: net/http.(*persistConn).readLoop(0x4000460120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 83 [select]: net/http.(*persistConn).writeLoop(0x4000460120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 99 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x4000208800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 100 [IO wait]: internal/poll.runtime_pollWait(0xffff8d472d60, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000208918, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000208900, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000208900, 0x0, 0x4000163698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x400046d908, 0x4000163700, 0x0, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x400046d908, 0x12ed9d0, 0x400046d908, 0xf78400, 0x4000142460) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x4000208880) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 101 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40024d0f70, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 102 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40024d0f70, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 103 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x40024d0f70) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 104 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x400249dda0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 114 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000130380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 107 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40002ec8c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 157 [sleep]: time.Sleep(0x12a05f200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/plasma.RunMemQuotaTunerIteration(0x40004cefb0, 0xee0d40, 0x4000250d70) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:211 +0x24c github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:247 +0xc0 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 goroutine 30 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x400257c700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 29 [semacquire]: sync.runtime_Semacquire(0x4004e58968) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004e58960) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x4004455b80, 0x400044d1d0, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffc6e2b969, 0x29, 0xffffc6e2b9cb, 0x25, 0x4004e26480, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004c51030, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffc6e2b969, 0x29, 0xffffc6e2b9cb, 0x25, 0x4004e26480, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004c51030, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004ca1190, 0x4004ca0e60, 0x400044d1d0, 0x400028dd40, 0x400023a800, 0x0, 0x1bf, 0x4002544da8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x400049a000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004c51030, 0xb, 0x4004c51028, 0x6, 0x4004c51040, 0xd, 0x4004c48900, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x400049a000, 0x4000410000, 0x400044ca20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x400049a000, 0x400022cd50, 0x2, 0x2, 0x400022cd60, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 24 [select]: net/http.(*persistConn).readLoop(0x40001c6360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 31 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x400257c700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 79 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400249b0c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 77 [select]: net/http.(*persistConn).readLoop(0x400028ed80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 78 [select]: net/http.(*persistConn).writeLoop(0x400028ed80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 80 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40014b6f00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 81 [IO wait]: internal/poll.runtime_pollWait(0xffff8d472b90, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40014c3218, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40014c3200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40014c3200, 0x58ec48, 0x40014b9c80, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c541e0, 0x4714f4, 0x40014b9c80, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c541e0, 0x4000490800, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x40014b78c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 130 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x400023b640, 0x40004acfc0, 0x40004ad030, 0x40047002f0, 0x12e8f90, 0x4000200620, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x4004c50344, 0x5, 0x40014b78c0, 0x12eca70, 0x40014b7780, 0x130d478, 0x40001f8150, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x4000200c40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x4000200c40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 87 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x4000200620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 88 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x4000200620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 89 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x4000157200, 0x40044d52c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 90 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x4000130080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 91 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x4000252be0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 92 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x4000252be0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 93 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x4000128538) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 94 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x4000252be0, 0x40044d5320) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 95 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400011c800, 0x1144559, 0x1b, 0x400245de38, 0x40044d5440, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004700810, 0x40044d5440, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400245df80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40044bef60, 0x4004700810) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 96 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 97 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x4000250870, 0xd00007, 0x400467f6d8, 0x5, 0x4004700820, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 131 [IO wait]: internal/poll.runtime_pollWait(0xffff8d472e48, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000132318, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000132300, 0x4002548000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000132300, 0x4002548000, 0x1000, 0x1000, 0xe, 0x4004c6c2a0, 0x23) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b80e8, 0x4002548000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c6360, 0x4002548000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400253e240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400253e240, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400253e240, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40024b9e30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40024b9e30, 0x4004bff200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40014b7b00, 0x4004bff200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40014b7b00, 0x4004bff200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40014b7b40, 0x4004bff200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004c4d080, 0x40044d5400, 0x4004768eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004c4d080, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004c4d080, 0xee0d40, 0x40014c0b90, 0x12db068, 0x40014b7b40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004d7c000, 0x40004df7a0, 0x4004d64420) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 146 [IO wait]: internal/poll.runtime_pollWait(0xffff8d4729c0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000266a18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000266a00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000266a00, 0x0, 0x4004d79d88, 0x665fe4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004729260, 0x611a6d7b, 0x4004d79db8, 0x445c24) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004729260, 0x4004d79e20, 0x4004d79e28, 0x18, 0x40024e7b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x40044587e0, 0x12ed9d0, 0x4004729260, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x40044587e0, 0x12ed9d0, 0x4004729260, 0x400049a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 147 [IO wait]: internal/poll.runtime_pollWait(0xffff8d4728d8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000266b18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000266b00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000266b00, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004729278, 0xffffb7b7ba68, 0xffff87df5038, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004729278, 0x665e1c, 0x0, 0xfe5e60, 0x40004c4dc8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004729290, 0x40047aa9c0, 0x11ae500, 0x4004458940, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x40044588c0, 0x12ec710, 0x4004729290, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x40044588c0, 0x12ec710, 0x4004729290, 0x400049a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 154 [IO wait]: internal/poll.runtime_pollWait(0xffff87dfe090, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000266e18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000266e00, 0x4004778000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000266e00, 0x4004778000, 0x1000, 0x1000, 0x40000a8be0, 0x469b10, 0x400476a180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000128598, 0x4004778000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004760240, 0x4004778000, 0x1000, 0x1000, 0x40000a8cf8, 0x67867c, 0x40044d5a40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40044bf560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40044bf560, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004760240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 155 [select]: net/http.(*persistConn).writeLoop(0x4004760240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 32 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x400257c700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 33 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x400257c700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 162 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x40001d9780) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 177 [select]: net/http.(*persistConn).writeLoop(0x4004698480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 174 [IO wait]: internal/poll.runtime_pollWait(0xffff8d472c78, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000133698, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000133680, 0x4004cdb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000133680, 0x4004cdb000, 0x1000, 0x1000, 0x4004cadbe0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b87f0, 0x4004cdb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c77a0, 0x4004cdb000, 0x1000, 0x1000, 0x4004cadcf8, 0x67867c, 0x4004cc6d20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004ccac00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004ccac00, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40001c77a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 165 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4002535ce0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 166 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4002535ce0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 167 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4002535ce0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 168 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x400257c700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 169 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400011c800, 0x1144559, 0x1b, 0x4002547e38, 0x4004cc6000, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004c84af0, 0x4004cc6000, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4002547f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x400253fda0, 0x4004c84af0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 175 [select]: net/http.(*persistConn).writeLoop(0x40001c77a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 172 [select]: net/http.(*persistConn).readLoop(0x40001c79e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 173 [select]: net/http.(*persistConn).writeLoop(0x40001c79e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 115 [IO wait]: internal/poll.runtime_pollWait(0xffff87dfe178, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000133798, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000133780, 0x4004c97000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000133780, 0x4004c97000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b87e8, 0x4004c97000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001c79e0, 0x4004c97000, 0x1000, 0x1000, 0x20, 0x30, 0x20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004ccab40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004ccab40, 0x40004fd90a, 0x2, 0x4, 0x40004fd9e0, 0x2, 0x40004cbc38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004ccab40, 0x7356e4, 0xffffc6e2b94e, 0xe, 0x4, 0x40004fda80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40045e56b0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40045e56b0, 0x4004694c00, 0x200, 0x200, 0xffffc6e2b94e, 0xe, 0x4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400249b8c0, 0x4004694c00, 0x200, 0x200, 0x40014a2f07, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400249b8c0, 0x4004694c00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400249b900, 0x4004694c00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004693760, 0x40004fdd00, 0x40004cbeb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004693760, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004693760, 0xee0d40, 0x4000180e10, 0x12db068, 0x400249b900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40046d41b0, 0x400047dce0, 0x40045eb800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 116 [select]: net/http.(*persistConn).roundTrip(0x40001c77a0, 0x400249bc40, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x40045e9900, 0xf8, 0x10a8660, 0x4004cd2e18) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x40045e9900, 0x1b6e900, 0xc03eb95cdcab9c6e, 0x1bfeeff07e) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x40045e9800, 0x12db048, 0x1b6e900, 0xc03eb95cdcab9c6e, 0x1bfeeff07e, 0x1be4a80, 0x0, 0x4000010430, 0x40045e5ce0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x40045e5cb0, 0x40045e9800, 0xc03eb95cdcab9c6e, 0x1bfeeff07e, 0x1be4a80, 0x4000010430, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x40045e5cb0, 0x40045e9800, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x400469a7b0, 0x1b, 0x4004cd3580, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x40046d47e0, 0x112f905, 0x6, 0x12db908, 0x400012d2e0, 0xee40c0, 0x40046d2618, 0x4004cd3580, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x400003e580, 0x46, 0x12db908, 0x400012d2e0, 0x400022bce0, 0x2, 0x4, 0x400022bd00, 0x2, 0x4, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x400003e580, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x20, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004cd3d40, 0x4, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x4004c88240, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x4004c88240, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001d97c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1267 +0xf8 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 117 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x4002535b80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:224 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 118 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x4002535b80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:612 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 119 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x4002535b80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:662 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 120 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x4002535b80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:752 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 121 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).processSchedIndexes(0x4002535b80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:296 +0x44 created by github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:218 +0x3c goroutine 122 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x40045e9a00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:249 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 196 [runnable]: net/http.(*persistConn).roundTrip(0x40001c7b00, 0x40001d9b80, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004ca8d00, 0xf8, 0x10a8660, 0x4004f14588) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004ca8d00, 0x1b6e900, 0xc03eb95cdcc814b5, 0x1bff0c68c5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004ca8c00, 0x12db048, 0x1b6e900, 0xc03eb95cdcc814b5, 0x1bff0c68c5, 0x1be4a80, 0x0, 0x40001b8818, 0x4004cf20c0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004cf2090, 0x4004ca8c00, 0xc03eb95cdcc814b5, 0x1bff0c68c5, 0x1be4a80, 0x40001b8818, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004cf2090, 0x4004ca8c00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004cc22a0, 0x1b, 0x4004f14cf0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004c8ce10, 0x112f905, 0x6, 0x12db908, 0x4000046e20, 0xee40c0, 0x4000133918, 0x4004f14cf0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x400003e600, 0x46, 0x12db908, 0x4000046e20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x400003e600, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x40047abfb0, 0xffff87df3f38) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004cfb4b0, 0x1063520, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x4004c886c0, 0x4004c886c0, 0x400003e600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.FetchNewClusterInfoCache(0xffffc6e2b94e, 0xe, 0x113075a, 0x7, 0x1130e44, 0x8, 0x4004ced0c4, 0x2, 0xf01) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:123 +0x134 github.com/couchbase/indexing/secondary/manager.(*IndexManager).FetchNewClusterInfoCache(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:306 github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).getIndexMetadata(0x1be5520, 0x12ec7a0, 0x40001d9b00, 0x400013b220, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:1284 +0x74 github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).handleIndexMetadataRequest(0x1be5520, 0x12edbe0, 0x400257c7e0, 0x4004ca8b00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:1271 +0x3e8 net/http.HandlerFunc.ServeHTTP(0x4004700710, 0x12edbe0, 0x400257c7e0, 0x4004ca8b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2069 +0x40 net/http.(*ServeMux).ServeHTTP(0x400023b900, 0x12edbe0, 0x400257c7e0, 0x4004ca8b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2448 +0x190 net/http.serverHandler.ServeHTTP(0x40044587e0, 0x12edbe0, 0x400257c7e0, 0x4004ca8b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2887 +0xbc net/http.(*conn).serve(0x4004cde140, 0x12f86a0, 0x40001d9980) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:1952 +0x71c created by net/http.(*Server).Serve /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:3013 +0x318 goroutine 124 [select]: net/http.(*persistConn).roundTrip(0x4004698480, 0x400249bc80, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x40045e9c00, 0xf8, 0x10a8660, 0x4002544dd8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x40045e9c00, 0x1b6e900, 0xc03eb95cdcc36260, 0x1bff07b670) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x40045e9b00, 0x12db048, 0x1b6e900, 0xc03eb95cdcc36260, 0x1bff07b670, 0x1be4a80, 0x0, 0x4000010458, 0x40046fc150, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x40046fc120, 0x40045e9b00, 0xc03eb95cdcc36260, 0x1bff07b670, 0x1be4a80, 0x4000010458, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x40046fc120, 0x40045e9b00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x400469a930, 0x1b, 0x4002545540, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x40046d4bd0, 0x112f905, 0x6, 0x12db908, 0x400012d360, 0xee40c0, 0x40046d2718, 0x4002545540, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x40014d2380, 0x46, 0x12db908, 0x400012d360, 0x4, 0x4000045e40, 0x2, 0x4, 0x4000045e60, 0x2, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x40014d2380, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x240, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4002545d00, 0x40046fc060, 0xffff8d493920) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x40046f0240, 0xd47600, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x40046f0240, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x400049a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9702 +0x344 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 125 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x400049a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1090 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 126 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x400049a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1017 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 195 [select]: net/http.(*persistConn).writeLoop(0x40046986c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 176 [IO wait]: internal/poll.runtime_pollWait(0xffff87dfdfa8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40046d2818, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40046d2800, 0x4004cdd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40046d2800, 0x4004cdd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b87f8, 0x4004cdd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4004698480, 0x4004cdd000, 0x1000, 0x1000, 0x60, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004ccad80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004ccad80, 0x1, 0x4004cc6e40, 0x43d584, 0x0, 0x0, 0x4000474568) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x4004698480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 129 [IO wait]: internal/poll.runtime_pollWait(0xffff87dfdcf0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40046d2a18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40046d2a00, 0x4004f18000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40046d2a00, 0x4004f18000, 0x1000, 0x1000, 0x0, 0x40046eb000, 0x4004cd5a38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010480, 0x4004f18000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).fill(0x40046ecc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40046ecc60, 0x40a, 0x4004cd5aa0, 0x0, 0xffff00001001, 0x400, 0x400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).ReadLine(0x40046ecc60, 0x1000, 0xffff8d4933d0, 0x1, 0x4004cd5bc8, 0x412ff8, 0x40046d5290) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:389 +0x30 net/textproto.(*Reader).readLineSlice(0x4004cd5cf0, 0x40046d5290, 0x4004cd5c48, 0xffff8d4933d0, 0x2c8, 0xffffb7b7b130) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/textproto/reader.go:57 +0xac net/textproto.(*Reader).ReadLine(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/textproto/reader.go:38 net/http.ReadResponse(0x40046ecc60, 0x4004f16000, 0x1000, 0x4004f18000, 0x4000010480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/response.go:161 +0x8c github.com/couchbase/cbauth/revrpc.(*Service).Run(0x40046fc450, 0x4004cd5f10, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:132 +0x2ac github.com/couchbase/cbauth/revrpc.BabysitService(0x4004cd5f10, 0x40046fc450, 0x0, 0x0, 0x0, 0xd8d03c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth/service.RegisterManager(0x13007d8, 0x40045e9a00, 0x0, 0x0, 0x40004cef80, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:146 +0xa0 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).registerWithServer(0x40045e9a00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:301 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:205 +0x78 goroutine 178 [select]: net/http.(*persistConn).roundTrip(0x40046986c0, 0x400249bcc0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x40045e9e00, 0xf8, 0x10a8660, 0x40004ce798) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x40045e9e00, 0x1b6e900, 0xc03eba6adcc51d84, 0x117740da194) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x40045e9d00, 0x12db048, 0x1b6e900, 0xc03eba6adcc51d84, 0x117740da194, 0x1be4a80, 0x0, 0x4000010470, 0x40046fc2d0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x40046fc2a0, 0x40045e9d00, 0xc03eba6adcc51d84, 0x117740da194, 0x1be4a80, 0x4000010470, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x40046fc2a0, 0x40045e9d00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x40046cfa40, 0x1f, 0x40004ced10, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/indexer.getWithAuth(0x40046cfa40, 0x1f, 0xe, 0x113846e, 0x11) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:3077 +0x68 github.com/couchbase/indexing/secondary/indexer.getGlobalTopology(0x40046ad270, 0xe, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:2266 +0x78 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).updateNodeList(0x40045e9a00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:220 +0x3c created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:206 +0x98 goroutine 194 [IO wait]: internal/poll.runtime_pollWait(0xffff87dfdec0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40046d2918, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40046d2900, 0x4004ce1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40046d2900, 0x4004ce1000, 0x1000, 0x1000, 0x4000059be8, 0x7ff1a8, 0x4000059cb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b8800, 0x4004ce1000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40046986c0, 0x4004ce1000, 0x1000, 0x1000, 0x60, 0x0, 0x40046cfa01) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004ccaea0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004ccaea0, 0x1, 0x4004cc6f60, 0x43d584, 0x0, 0xd8c91c, 0x4000059d68) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40046986c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 197 [IO wait]: internal/poll.runtime_pollWait(0xffff87dfddd8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000133818, 0x72, 0x0, 0x1, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000133800, 0x40047abe41, 0x1, 0x1, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000133800, 0x40047abe41, 0x1, 0x1, 0x4000000001, 0x0, 0x4004cca7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001b8808, 0x40047abe41, 0x1, 0x1, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*connReader).backgroundRead(0x40047abe30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:692 +0x50 created by net/http.(*connReader).startBackgroundRead /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:688 +0xc4 goroutine 200 [select]: net/http.(*persistConn).readLoop(0x40001c7b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 201 [select]: net/http.(*persistConn).writeLoop(0x40001c7b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc Initializing write barrier = 8000 2021-08-16T13:52:00.716+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:52:00.716+00:00 [Info] Setting ipv6=false 2021-08-16T13:52:00.721+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:52:00.729+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:52:00.729+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:52:00.729+00:00 [Info] Setting log level to Info 2021-08-16T13:52:00.729+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:52:00.732+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:52:00 audit: created new audit service 2021-08-16T13:52:00.740+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:52:00.740+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:52:00.740+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:52:00.740+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:52:00.740+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:52:00.752+00:00 [Info] security context initialized 2021-08-16T13:52:00.752+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:52:00.752+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:52:00.752+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:52:00.752+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:52:00.752+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:52:00.765+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:52:00.767+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:52:00.780+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:52:00.799+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:52:00.803+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:52:00.804+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:52:00.805+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:52:00.808+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:52:00.809+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:52:00.810+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:52:00.811+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:52:00.854+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:52:00.854+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 50 us 2021-08-16T13:52:00.855+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:52:00.857+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:52:00.857+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:52:00.858+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:52:00.858+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:52:00.858+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:52:00.858+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:52:00.858+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:52:00.859+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:52:00.860+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:52:00.861+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:52:00.861+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:52:00.861+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:52:00.861+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:52:00.862+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:52:00.862+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:52:00.862+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:52:00.862+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:52:00.862+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:52:00.862+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:52:00.862+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:52:00.867+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 5.164556ms 2021-08-16T13:52:00.868+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:52:00.871+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:52:00.871+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:52:00.871+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:52:00.871+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:52:00.871+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:52:00.871+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... unexpected fault address 0x40001d9b1f fatal error: fault 2021-08-16T13:52:00.882+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:52:00.882+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:52:00.882+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:52:00.882+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:52:00.882+00:00 [Info] : Recover shard shard1 from metadata completed. [signal SIGBUS: bus error code=0x1 addr=0x40001d9b1f pc=0xbd8ea4] goroutine 175 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004c729f0 sp=0x4004c729c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004c72a30 sp=0x4004c729f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x400045d520, 0x4004457800, 0x11a9890, 0x4004c72b40, 0x400045d538, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004c72ab0 sp=0x4004c72a40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x400045d520, 0x4004457800, 0x11a9890, 0x4004c72b40, 0x400045d538, 0xffffbdb555e0, 0x4004c72b78, 0x412ddc) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004c72af0 sp=0x4004c72ab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4000032b40, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004c72da0 sp=0x4004c72af0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4000032b40, 0x400007f3b0, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004c72e60 sp=0x4004c72da0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004b771e0, 0x4004b771f0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004c73200 sp=0x4004c72e60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004b771e0, 0x4004b771f0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004c738f0 sp=0x4004c73200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004ed4ff0, 0x4004ed8a80, 0x4004eba580, 0x4004b771c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004c73fb0 sp=0x4004c738f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004c73fb0 sp=0x4004c73fb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x4000220000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x4000286fc0, 0x4, 0x1136a87, 0xf, 0x40000f9e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 7 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400000e690) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 8 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400000e6a8) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 9 [IO wait]: internal/poll.runtime_pollWait(0xffff94898888, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0d98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0d80, 0x4000137000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0d80, 0x4000137000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011e000, 0x4000137000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x4000142000, 0x40004a6001, 0xdff, 0xdff, 0x400009b8e8, 0x6902d8, 0x40000f5948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x4000140018, 0x40004a6001, 0xdff, 0xdff, 0x40000f59f8, 0x6f1968, 0x40000f5938) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40000cef20, 0xa, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40000cef20, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40000cef20, 0xf5a960, 0x4000429580, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4000429560, 0x4000046860, 0x40000f5b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400008cbe0, 0x12f8f98, 0x4000429560, 0x4000037757, 0x16, 0x4000455680, 0x400009bc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400008cbe0, 0x12f8f98, 0x4000429560, 0x400008cbe0, 0x4000036f68, 0x4000036f70, 0x40000f0e00, 0x4000046780, 0xee0980, 0x4000455680, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400008cbe0, 0x12f8f98, 0x4000429560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4000097f20, 0x400009bf78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x4000267f78, 0x4000097f20, 0x12db8a8, 0x400000ed20, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x4000097f20, 0x40000b1360, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x4000097f20, 0x40000b1360) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 10 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 11 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 12 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 13 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 14 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 15 [syscall]: syscall.Syscall(0x3f, 0x0, 0x40000547a0, 0x4, 0x4000054668, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x40000547a0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 18 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400000ac00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 24 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000226900) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 54 [IO wait]: internal/poll.runtime_pollWait(0xffff948987a0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000262118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000262100, 0x40004a7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000262100, 0x40004a7000, 0x1000, 0x1000, 0x4000067a88, 0x68f7a4, 0x40001349f0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011e020, 0x40004a7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40004546c0, 0x40004a7000, 0x1000, 0x1000, 0xf12180, 0x4000254130, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400021c360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400021c360, 0xa, 0x1008880, 0x40002540f0, 0x199, 0xad9, 0x4000067b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400021c360, 0x2, 0x0, 0x9, 0x56b, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400010e720) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400010e720, 0x400026f000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000138200, 0x400026f000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000138200, 0x400026f000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000138240, 0x400026f000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x40001349a0, 0x0, 0x4000067eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40001349a0, 0x0, 0x0, 0x4000067f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40001349a0, 0xee0d40, 0x40002540f0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40001245a0, 0x400021c720, 0x400021ec00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 52 [select]: net/http.(*persistConn).readLoop(0x40004546c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 53 [select]: net/http.(*persistConn).writeLoop(0x40004546c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 66 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400029c2e0, 0x12c18bc, 0x1, 0x400026be58, 0x400021e6c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x40001a46c8, 0x400021e6c0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400026bfa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4000226000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 67 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4000226000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 76 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000457200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 70 [IO wait]: internal/poll.runtime_pollWait(0xffff948986b8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400015c098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400015c080, 0x4000166000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400015c080, 0x4000166000, 0x1000, 0x1000, 0x4000163be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010378, 0x4000166000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c7e0, 0x4000166000, 0x1000, 0x1000, 0x4000163cf8, 0x67867c, 0x40001224e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000142540) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000142540, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000c7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 71 [select]: net/http.(*persistConn).writeLoop(0x400000c7e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 93 [IO wait]: internal/poll.runtime_pollWait(0xffff94898148, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400015d818, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400015d800, 0x40023f8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400015d800, 0x40023f8000, 0x1000, 0x1000, 0x4000064be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010730, 0x40023f8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001fe900, 0x40023f8000, 0x1000, 0x1000, 0x4000064cf8, 0x67867c, 0x4000038420) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40024ef980) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40024ef980, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40001fe900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 23 [select]: net/http.(*persistConn).writeLoop(0x4000455200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 38 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000080000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 37 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x400019a620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 58 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x400019a620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 40 [IO wait]: internal/poll.runtime_pollWait(0xffff94897f78, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400015cf18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400015cf00, 0x40024d7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400015cf00, 0x40024d7000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010bd0, 0x40024d7000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000412a20, 0x40024d7000, 0x1000, 0x1000, 0x4000206900, 0x200000003, 0x4000206900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40024ee1e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40024ee1e0, 0x400027a00a, 0x40002698c8, 0x4d3b10, 0x40000820c0, 0x400027a000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40024ee1e0, 0x0, 0x1, 0x1c211e8, 0x40000820d8, 0x400027a000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400441a6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400441a6c0, 0x40044d8000, 0x1000, 0x1000, 0x40002699b8, 0x44d8a8, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000081a40, 0x40044d8000, 0x1000, 0x1000, 0x400018ec58, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000081a40, 0x40044d8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000081a80, 0x40044d8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000269dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000269dd0, 0xa, 0x40044d8201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000269dd0, 0x40044d820a, 0x1, 0x4000111dfc, 0x0, 0x40044d8263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000269dd0, 0x400004600a, 0x4000111dfc, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400013ce10, 0x40002008d0, 0x24, 0x4000269f20, 0x400245ab58, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x400013ce10, 0x113075a, 0x7, 0x400245ab58, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x400013cdd0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 39 [IO wait]: internal/poll.runtime_pollWait(0xffff94898060, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000262298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000262280, 0x4000217000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000262280, 0x4000217000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011e0e0, 0x4000217000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000454c60, 0x4000217000, 0x1000, 0x1000, 0x4000206a80, 0x200000003, 0x4000206a80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400018ef00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400018ef00, 0x400027a00a, 0x40001ef8c8, 0x4d3b10, 0x40000820c0, 0x400027a000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400018ef00, 0x0, 0x1, 0x0, 0x40000820d8, 0x400027a000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40024733b0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40024733b0, 0x400447f000, 0x1000, 0x1000, 0x40001ef9b8, 0x0, 0x40001ef988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000138940, 0x400447f000, 0x1000, 0x1000, 0x4000102a80, 0x3, 0x40001efbe0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000138940, 0x400447f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000138980, 0x400447f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40001efdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001efdd0, 0xa, 0x400447f201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40001efdd0, 0x400447f20a, 0x1, 0x4000111df9, 0x0, 0x400447f2e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40001efdd0, 0x400013d20a, 0x4000111df9, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400013ce10, 0x4000272660, 0x17, 0x40001eff20, 0x400245ab70, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x400013ce10, 0x113075a, 0x7, 0x400245ab70, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x400013cdd0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 97 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40044f8440) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 81 [select]: net/http.(*persistConn).readLoop(0x4000412a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 59 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x400019a620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 60 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x400019a620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 82 [select]: net/http.(*persistConn).writeLoop(0x4000412a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 45 [IO wait]: internal/poll.runtime_pollWait(0xffff94897e90, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400015d498, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400015d480, 0x4000476000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400015d480, 0x4000476000, 0x1000, 0x1000, 0x400026d688, 0x69f178, 0x400026d6f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010630, 0x4000476000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001fe360, 0x4000476000, 0x1000, 0x1000, 0x400026d798, 0x431720, 0x400026d770) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40024eef60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40024eef60, 0xa, 0x400026d838, 0x432048, 0xffff943f2328, 0x400026d8a0, 0x28) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40024eef60, 0x413180, 0x400026d9a0, 0xffff943f2350, 0x400026d8a0, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40024734a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40024734a0, 0x400024b000, 0x1000, 0x1000, 0x400026d998, 0x44e884, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000138a00, 0x400024b000, 0x1000, 0x1000, 0x400018ec58, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000138a00, 0x400024b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000138a40, 0x400024b000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400026dd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400026dd80, 0xa, 0x400024b201, 0x1, 0xffff946c5e20, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400026dd80, 0x400024b20a, 0x1, 0x4002497840, 0x0, 0x400024b259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400026dd80, 0x40000cf60a, 0x4002497840, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x400013ce10, 0x4000464280, 0x1f, 0x400026dee0, 0x40004662d0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x400013ce10, 0x113075a, 0x7, 0x4000111e10, 0xd, 0x40004662d0, 0x0, 0x4714f4, 0x400013ce10) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x400013cdd0, 0x4000111e10, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 43 [select]: net/http.(*persistConn).readLoop(0x4000454c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 44 [select]: net/http.(*persistConn).writeLoop(0x4000454c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 85 [select]: net/http.(*persistConn).readLoop(0x40001fe360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 86 [select]: net/http.(*persistConn).writeLoop(0x40001fe360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 25 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x40044f85c0, 0x4004423670) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 94 [select]: net/http.(*persistConn).writeLoop(0x40001fe900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 61 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x40004297a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 62 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x40004a2240) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 63 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x40000f0200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 64 [IO wait]: internal/poll.runtime_pollWait(0xffff94897cc0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0418, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000f0400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000f0400, 0x0, 0x4000053e98, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x400439c4c8, 0x4000053f00, 0x0, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x400439c4c8, 0x12ed9d0, 0x400439c4c8, 0xf78400, 0x40000b1360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x40000f0300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 65 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40044d45b0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 98 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40044d45b0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 99 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x40044d45b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 100 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x40004a3620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 101 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40000e01c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 118 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x400019ac40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 22 [select]: net/http.(*persistConn).readLoop(0x4000455200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 102 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000457240) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 48 [IO wait]: internal/poll.runtime_pollWait(0xffff94897da8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000263198, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000263180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000263180, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004b975f0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004b975f0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x4004ba60c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 49 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x40044f85c0, 0x4000275340, 0x40002753b0, 0x4004423670, 0x12e8f90, 0x400470a000, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x4004b68204, 0x5, 0x4004ba60c0, 0x12eca70, 0x4000457980, 0x130d478, 0x40002fa770, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x400019ab60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x400019ab60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 114 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x40002974a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 115 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x40002974a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 130 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x400470a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 131 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x400470a000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 132 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x40044ae900, 0x4004692f00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 133 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x40004571c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 134 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x400013f2c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 135 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x400013f2c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 136 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x400011e4b0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 137 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x400013f2c0, 0x4004692f60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 138 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400029c2e0, 0x1144559, 0x1b, 0x4004bcfe38, 0x4004693080, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004b9cd40, 0x4004693080, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004bcff80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40045dacc0, 0x4004b9cd40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 139 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 140 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x4000254be0, 0xd00007, 0x4004bb8940, 0x5, 0x4004b9cd50, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 141 [IO wait]: internal/poll.runtime_pollWait(0xffff94897af0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000263318, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000263300, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000263300, 0xffffbdb555b8, 0x30, 0x4004b550e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004b97ae8, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004b97ae8, 0x4004b550e0, 0x11ae500, 0x4004450da0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x4004450d20, 0x12ed9d0, 0x4004b97ae8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x4004450d20, 0x12ed9d0, 0x4004b97ae8, 0x4000220000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 142 [IO wait]: internal/poll.runtime_pollWait(0xffff94897a08, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000263418, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000263400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000263400, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004b97b00, 0xffffbdb555b8, 0xffff9482caa8, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004b97b00, 0x665e1c, 0x0, 0xfe5e60, 0x4004bc15c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004b97b18, 0x4004b55170, 0x11ae500, 0x4004450e80, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x4004450e00, 0x12ec710, 0x4004b97b18, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x4004450e00, 0x12ec710, 0x4004b97b18, 0x4000220000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 103 [IO wait]: internal/poll.runtime_pollWait(0xffff94897bd8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000262498, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000262480, 0x40044db000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000262480, 0x40044db000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001940a8, 0x40044db000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000455200, 0x40044db000, 0x1000, 0x1000, 0x20, 0x30, 0x20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400021cf60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400021cf60, 0x100a, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400021cf60, 0x7356e4, 0xffffcb1c294e, 0xe, 0x40001cf4a8, 0x40b2f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004618c60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004618c60, 0x4004609c00, 0x200, 0x200, 0xffffcb1c294e, 0xe, 0x4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000227940, 0x4004609c00, 0x200, 0x200, 0x4000038420, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000227940, 0x4004609c00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000227980, 0x4004609c00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004c38f20, 0x680d00, 0x40001cf6b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004c38f20, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004c38f20, 0xee0d40, 0x4002404820, 0x12db068, 0x4000227980) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004678a20, 0x40046746c0, 0x4004676840) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 116 [semacquire]: sync.runtime_Semacquire(0x4004ed4ff8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004ed4ff0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x4004eba580, 0x40004a2780, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffcb1c2969, 0x29, 0xffffcb1c29cb, 0x25, 0x4004b6bce0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004b89dd0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffcb1c2969, 0x29, 0xffffcb1c29cb, 0x25, 0x4004b6bce0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004b89dd0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004c73190, 0x4004c72e60, 0x40004a2780, 0x400021ad80, 0x4000457200, 0x0, 0xbf, 0x4000470da8, 0x412c2c, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x4000220000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004b89dd0, 0xb, 0x4004b89dc8, 0x6, 0x4004b89de0, 0xd, 0x4004b95d40, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x4000220000, 0x4000420000, 0x4000202060) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x4000220000, 0x40001123e0, 0x2, 0x2, 0x40001123f0, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 144 [IO wait]: internal/poll.runtime_pollWait(0xffff94659688, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000263718, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000263700, 0x4004ba3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000263700, 0x4004ba3000, 0x1000, 0x1000, 0x4000161be0, 0x469b10, 0x4000103e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011e4d8, 0x4004ba3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400012b8c0, 0x4004ba3000, 0x1000, 0x1000, 0x4000161cf8, 0x67867c, 0x40046933e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40045daf60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40045daf60, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400012b8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 145 [select]: net/http.(*persistConn).writeLoop(0x400012b8c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 119 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x400019ac40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 117 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x400019ac40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 120 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x400019ac40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 121 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4000457d80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 122 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x400014dce0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 123 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x400014dce0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 124 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x400014dce0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 125 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x400019ac40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 126 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400029c2e0, 0x1144559, 0x1b, 0x40001ebe38, 0x4004b6aba0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004b769f0, 0x4004b6aba0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x40001ebf80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004b66a20, 0x4004b769f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 163 [IO wait]: internal/poll.runtime_pollWait(0xffff94659770, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004e98098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004e98080, 0x4004b6f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004e98080, 0x4004b6f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010a60, 0x4004b6f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001ff560, 0x4004b6f000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004b67440) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004b67440, 0xa, 0x0, 0x0, 0x0, 0x0, 0x4004e87438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004b67440, 0x0, 0x4004e87478, 0x655784, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004b55d10) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004b55d10, 0x4004b72a00, 0x200, 0x200, 0x4, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000457ec0, 0x4004b72a00, 0x200, 0x200, 0x4004b74307, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000457ec0, 0x4004b72a00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000457f00, 0x4004b72a00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004b7e840, 0x0, 0x4004e876b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004b7e840, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004b7e840, 0xee0d40, 0x400008d4f0, 0x12db068, 0x4000457f00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004e88480, 0x4004b675c0, 0x4004b6b6e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 129 [select]: net/http.(*persistConn).readLoop(0x40001ff560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 162 [select]: net/http.(*persistConn).writeLoop(0x40001ff560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 164 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000457dc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 165 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x400014db80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 166 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x400014db80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 167 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x400014db80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 168 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x400014db80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 169 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004b63100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 170 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x4004b63100, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 171 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x4000220000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 172 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x4000220000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 173 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x4000220000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 174 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 Initializing write barrier = 8000 2021-08-16T13:52:06.047+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:52:06.047+00:00 [Info] Setting ipv6=false 2021-08-16T13:52:06.060+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:52:06.062+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:52:06.062+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:52:06.062+00:00 [Info] Setting log level to Info 2021-08-16T13:52:06.063+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:52:06.065+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:52:06 audit: created new audit service 2021-08-16T13:52:06.073+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:52:06.073+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:52:06.074+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:52:06.074+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:52:06.074+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:52:06.084+00:00 [Info] security context initialized 2021-08-16T13:52:06.084+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:52:06.084+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:52:06.084+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:52:06.084+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:52:06.084+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:52:06.097+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:52:06.107+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:52:06.111+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:52:06.125+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:52:06.126+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:52:06.126+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:52:06.128+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:52:06.128+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:52:06.130+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:52:06.130+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:52:06.131+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:52:06.176+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:52:06.177+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 57 us 2021-08-16T13:52:06.177+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:52:06.179+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:52:06.180+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:52:06.181+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:52:06.181+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:52:06.181+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:52:06.181+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:52:06.181+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:52:06.183+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:52:06.183+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:52:06.185+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:52:06.185+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:52:06.185+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:52:06.185+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:52:06.185+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:52:06.185+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:52:06.185+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:52:06.185+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:52:06.185+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:52:06.185+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:52:06.185+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:52:06.190+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 4.841121ms 2021-08-16T13:52:06.190+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:52:06.191+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:52:06.191+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:52:06.191+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:52:06.191+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:52:06.191+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:52:06.191+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:52:06.202+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:52:06.202+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:52:06.202+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:52:06.202+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:52:06.203+00:00 [Info] RebalanceMgr::initService Init 2021-08-16T13:52:06.203+00:00 [Info] ServiceMgr::registerWithServer nodeuuid 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:52:06.203+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x4004e0e49f fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x4004e0e49f pc=0xbd8ea4] goroutine 190 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x400457a9f0 sp=0x400457a9c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x400457aa30 sp=0x400457a9f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x400024f860, 0x4004752c00, 0x11a9890, 0x400457ab40, 0x400024f878, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x400457aab0 sp=0x400457aa40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x400024f860, 0x4004752c00, 0x11a9890, 0x400457ab40, 0x400024f878, 0xc18f8c, 0x400457ac18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x400457aaf0 sp=0x400457aab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4000250a00, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x400457ada0 sp=0x400457aaf0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4000250a00, 0x40023675e0, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x400457ae60 sp=0x400457ada0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004677fc0, 0x4004677fd0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x400457b200 sp=0x400457ae60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004677fc0, 0x4004677fd0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x400457b8f0 sp=0x400457b200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004e730c0, 0x400477fc00, 0x40000b9b80, 0x4004677fa0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x400457bfb0 sp=0x400457b8f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x400457bfb0 sp=0x400457bfb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x4000182700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x400008d4d0, 0x4, 0x1136a87, 0xf, 0x4000199e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 5 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x400000e120) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 6 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x400000e138) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 7 [IO wait]: internal/poll.runtime_pollWait(0xffff7951ae88, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400020e198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400020e180, 0x4000235000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400020e180, 0x4000235000, 0x1000, 0x1000, 0x1311860, 0x47b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000206028, 0x4000235000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x40002001e0, 0x4000246001, 0xdff, 0xdff, 0x400011f8e8, 0x6902d8, 0x4000195948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x4000216060, 0x4000246001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x4000232160, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4000232160, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4000232160, 0xf5a960, 0x40002002c0, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x40002002a0, 0x400020a200, 0x4000195b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x4000240000, 0x12f8f98, 0x40002002a0, 0x400020c907, 0x16, 0x4000228480, 0x400011fc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x4000240000, 0x12f8f98, 0x40002002a0, 0x4000240000, 0x400020c110, 0x400020c120, 0x400020e200, 0x400020a120, 0xee0980, 0x4000228480, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x4000240000, 0x12f8f98, 0x40002002a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x400008c450, 0x400011ff78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x4000125f78, 0x400008c450, 0x12db8a8, 0x4000216030, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x400008c450, 0x4000098460, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x400008c450, 0x4000098460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 129 [IO wait]: internal/poll.runtime_pollWait(0xffff7868a980, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400020fe18, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400020fe00, 0x4004c6e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400020fe00, 0x4004c6e000, 0x1000, 0x1000, 0x40001b6be0, 0x469b10, 0x400257a900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010510, 0x4004c6e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001eafc0, 0x4004c6e000, 0x1000, 0x1000, 0x40001b6cf8, 0x67867c, 0x400469fce0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004c6c0c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004c6c0c0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40001eafc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 8 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 9 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 10 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 11 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 12 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 20 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x400010c300) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 13 [syscall]: syscall.Syscall(0x3f, 0x0, 0x4000427fa0, 0x4, 0x4000427e68, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x4000427fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000140000, 0x4000427fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x400013e000, 0x4000427fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 52 [IO wait]: internal/poll.runtime_pollWait(0xffff7951ada0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000190198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000190180, 0x4002360000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000190180, 0x4002360000, 0x1000, 0x1000, 0x4000069a88, 0x68f7a4, 0x4000232470) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400235e000, 0x4002360000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400014f320, 0x4002360000, 0x1000, 0x1000, 0xf12180, 0x400014a040, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40023580c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40023580c0, 0xa, 0x1008880, 0x400014a000, 0x199, 0xad9, 0x4000069b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40023580c0, 0x2, 0x0, 0x9, 0x56b, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40002023c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40002023c0, 0x4002361000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000210100, 0x4002361000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000210100, 0x4002361000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40002101c0, 0x4002361000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4000232420, 0x0, 0x4000069eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4000232420, 0x0, 0x0, 0x4000069f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4000232420, 0xee0d40, 0x400014a000, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40000a6000, 0x4000200420, 0x400010eb40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 66 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40024681c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 50 [select]: net/http.(*persistConn).readLoop(0x400014f320) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 51 [select]: net/http.(*persistConn).writeLoop(0x400014f320) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 36 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400006c710, 0x12c18bc, 0x1, 0x4000121e58, 0x400010e7e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x40004226c8, 0x400010e7e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4000121fa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4000161600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 37 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4000161600) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 82 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x4000188460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 40 [IO wait]: internal/poll.runtime_pollWait(0xffff7951acb8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400020e118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400020e100, 0x4000234000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400020e100, 0x4000234000, 0x1000, 0x1000, 0x4000019be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40002060d0, 0x4000234000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40002285a0, 0x4000234000, 0x1000, 0x1000, 0x4000019cf8, 0x67867c, 0x400027e360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002358480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4002358480, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40002285a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 41 [select]: net/http.(*persistConn).writeLoop(0x40002285a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 115 [IO wait]: internal/poll.runtime_pollWait(0xffff7951a2c0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400020fc18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x400020fc00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x400020fc00, 0x12d9d48, 0x4000161b40, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c01038, 0x0, 0x12ed9d0, 0x4002572528) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c01038, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x40002111c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 114 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4002469240) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 146 [select]: net/http.(*persistConn).writeLoop(0x40001eafc0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 61 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).processSchedIndexes(0x4004b70210) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:296 +0x44 created by github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:218 +0x3c goroutine 62 [IO wait]: internal/poll.runtime_pollWait(0xffff7868a6c8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004499718, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004499700, 0x4004dff000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004499700, 0x4004dff000, 0x1000, 0x1000, 0x0, 0x4004dfe000, 0x4004df3a38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400235e2d8, 0x4004dff000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).fill(0x4004df81e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004df81e0, 0x654f0a, 0x4004df3b08, 0x439548, 0x4004df3af8, 0x5429e4, 0x4004499700) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).ReadLine(0x4004df81e0, 0x655058, 0x4004df3b08, 0x655070, 0x4004df3bc8, 0x412ff8, 0x4004d61a70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:389 +0x30 net/textproto.(*Reader).readLineSlice(0x4004df3cf0, 0x4004d61a70, 0x4004df3c48, 0x6567e4, 0x187bbb0c3bf406f8, 0xf26100) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/textproto/reader.go:57 +0xac net/textproto.(*Reader).ReadLine(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/textproto/reader.go:38 net/http.ReadResponse(0x4004df81e0, 0x4004dfc000, 0x1000, 0x4004dff000, 0x400235e2d8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/response.go:161 +0x8c github.com/couchbase/cbauth/revrpc.(*Service).Run(0x4004ddc750, 0x4004df3f10, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:132 +0x2ac github.com/couchbase/cbauth/revrpc.BabysitService(0x4004df3f10, 0x4004ddc750, 0x0, 0x0, 0x0, 0xd8d03c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth/service.RegisterManager(0x13007d8, 0x4004b55500, 0x0, 0x0, 0x400001bf80, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/service/revrpc.go:146 +0xa0 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).registerWithServer(0x4004b55500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:301 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:205 +0x78 goroutine 23 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x4000188460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 24 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x4000188460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 25 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x4000188460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 53 [select]: net/http.(*persistConn).readLoop(0x40002299e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 54 [select]: net/http.(*persistConn).writeLoop(0x40002299e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 68 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40024685c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 69 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x400249e900) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 70 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x400246e420) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 71 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x40000aa080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 85 [IO wait]: internal/poll.runtime_pollWait(0xffff7951a490, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400020e898, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400020e880, 0x40024d4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400020e880, 0x40024d4000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010088, 0x40024d4000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000228120, 0x40024d4000, 0x1000, 0x1000, 0x4000001b00, 0x200000003, 0x4000001b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4002358e40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4002358e40, 0x400017000a, 0x40024d38c8, 0x4d3b10, 0x4000140060, 0x4000170000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4002358e40, 0x0, 0x1, 0x0, 0x4000140078, 0x4000170000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400235d230) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400235d230, 0x40024d6000, 0x1000, 0x1000, 0x40024d39b8, 0x44d8a8, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000161a40, 0x40024d6000, 0x1000, 0x1000, 0x40023589b8, 0x106f080, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000161a40, 0x40024d6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000161a80, 0x40024d6000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40024d3dd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40024d3dd0, 0xa, 0x40024d6201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40024d3dd0, 0x40024d620a, 0x1, 0x40000371d9, 0x0, 0x40024d62e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40024d3dd0, 0x400018560a, 0x40000371d9, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000184380, 0x400003a4c8, 0x17, 0x40024d3f20, 0x40024be330, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x4000184380, 0x113075a, 0x7, 0x40024be330, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x4000184340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 86 [IO wait]: internal/poll.runtime_pollWait(0xffff7951a578, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400020f918, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400020f900, 0x40000bd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400020f900, 0x40000bd000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010008, 0x40000bd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40002299e0, 0x40000bd000, 0x1000, 0x1000, 0x400025aa80, 0x200000003, 0x400025aa80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000200540) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000200540, 0x400017000a, 0x40001cb8c8, 0x4d3b10, 0x4000140060, 0x4000170000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000200540, 0x0, 0x1, 0x0, 0x4000140078, 0x4000170000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400246fcb0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400246fcb0, 0x400257d000, 0x1000, 0x1000, 0x40001cb9b8, 0x0, 0x40001cb988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4002468480, 0x400257d000, 0x1000, 0x1000, 0x4000102d80, 0x3, 0x40001cbbe0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4002468480, 0x400257d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40024684c0, 0x400257d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x40001cbdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40001cbdd0, 0xa, 0x400257d201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x40001cbdd0, 0x400257d20a, 0x1, 0x40024bb03b, 0x0, 0x400257d263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x40001cbdd0, 0x400020a30a, 0x40024bb03b, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000184380, 0x40000427e0, 0x24, 0x40001cbf20, 0x40024be318, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x4000184380, 0x113075a, 0x7, 0x40024be318, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x4000184340) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 72 [IO wait]: internal/poll.runtime_pollWait(0xffff7951a660, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000aa198, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000aa180, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000aa180, 0x0, 0x4000054698, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4002572528, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4002572528, 0x12ed9d0, 0x4002572528, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x40000aa100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 73 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400248a8f0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 74 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x400248a8f0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 75 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x400248a8f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 76 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x400246f800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 109 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000210940) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 91 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40000fa620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 89 [select]: net/http.(*persistConn).readLoop(0x4000228120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 90 [select]: net/http.(*persistConn).writeLoop(0x4000228120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 79 [select]: net/http.(*persistConn).readLoop(0x400014e360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 80 [select]: net/http.(*persistConn).writeLoop(0x400014e360) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 32 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x4000188c40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 92 [IO wait]: internal/poll.runtime_pollWait(0xffff7951a748, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x400020eb98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x400020eb80, 0x40024d8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x400020eb80, 0x40024d8000, 0x1000, 0x1000, 0x4000123688, 0x69f178, 0x40001236f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010158, 0x40024d8000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000228900, 0x40024d8000, 0x1000, 0x1000, 0x5, 0x40024de000, 0x40001237c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40000901e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40000901e0, 0x68f90a, 0x40000d96b0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40000901e0, 0x0, 0x4000123868, 0x41a0a8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400235d590) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400235d590, 0x40024da000, 0x1000, 0x1000, 0x4000123998, 0x44e884, 0x1c00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000161b80, 0x40024da000, 0x1000, 0x1000, 0x4000200898, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000161b80, 0x40024da000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000161bc0, 0x40024da000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x4000123d80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000123d80, 0xa, 0x40024da201, 0x1, 0xffff7893dfe0, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x4000123d80, 0x40024da20a, 0x1, 0x400247ef10, 0x0, 0x40024da259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x4000123d80, 0x4000232d0a, 0x400247ef10, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000184380, 0x40024c0420, 0x1f, 0x4000123ee0, 0x40024be810, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x4000184380, 0x113075a, 0x7, 0x40000371f0, 0xd, 0x40024be810, 0x0, 0x0, 0x2) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x4000184340, 0x40000371f0, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 97 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40024689c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 95 [select]: net/http.(*persistConn).readLoop(0x4000228900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 96 [select]: net/http.(*persistConn).writeLoop(0x4000228900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 33 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x4000188c40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 28 [IO wait]: internal/poll.runtime_pollWait(0xffff7951a1d8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000190418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000190400, 0x40044ef000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000190400, 0x40044ef000, 0x1000, 0x1000, 0x40001b9be0, 0x469b10, 0x4000103980) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400013e510, 0x40044ef000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000cc60, 0x40044ef000, 0x1000, 0x1000, 0x40001b9cf8, 0x67867c, 0x4000039080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003e3560) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40003e3560, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000cc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 29 [select]: net/http.(*persistConn).writeLoop(0x400000cc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 116 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x4000211600, 0x4000476ee0, 0x4000476f50, 0x4004c04a60, 0x12e8f90, 0x400253a380, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x40046d8db4, 0x5, 0x40002111c0, 0x12eca70, 0x4000211080, 0x130d478, 0x40000fa690, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40024c6700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40024c6700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 117 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x400253a380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 118 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x400253a380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 119 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x400017ee10, 0x400469f9e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 120 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x4002468980) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 121 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x40044090e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 122 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x40044090e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 123 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x40000104e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 124 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x4004409180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 125 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x4004409180) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 126 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x4000211600, 0x4004c04a60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 110 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x40044090e0, 0x400456e0c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 111 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400006c710, 0x1144559, 0x1b, 0x400011fe38, 0x400456e1e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x40045386c0, 0x400456e1e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x400246ff50) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400011ff80, 0x0, 0x100000001) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x400453a660, 0x40045386c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 112 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 113 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x4000240be0, 0xd00007, 0x40045644e8, 0x5, 0x40045386d0, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 130 [IO wait]: internal/poll.runtime_pollWait(0xffff7951a0f0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000ab218, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000ab200, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000ab200, 0x0, 0x40001bbd88, 0x665fe4) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x400456c438, 0x611a6d86, 0x40001bbdb8, 0x445c24) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x400456c438, 0x40001bbe20, 0x40001bbe28, 0x18, 0x400257b080) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x400253a540, 0x12ed9d0, 0x400456c438, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x400253a540, 0x12ed9d0, 0x400456c438, 0x4000182700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 131 [IO wait]: internal/poll.runtime_pollWait(0xffff7951a008, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000ab318, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40000ab300, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40000ab300, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x400456c450, 0xffffa3c38f18, 0xffff7a9ac810, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x400456c450, 0x665e1c, 0x0, 0xfe5e60, 0x4004c3d5c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x400456c468, 0x400453e9c0, 0x11ae500, 0x400253a6a0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x400253a620, 0x12ec710, 0x400456c468, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x400253a620, 0x12ec710, 0x400456c468, 0x4000182700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 127 [IO wait]: internal/poll.runtime_pollWait(0xffff7951a3a8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000aa298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000aa280, 0x4002577000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000aa280, 0x4002577000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40002061b0, 0x4002577000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400014e360, 0x4002577000, 0x1000, 0x1000, 0x20, 0x30, 0x20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400249ed80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400249ed80, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400249ed80, 0x97ee90, 0x4000476d90, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004c0d590) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004c0d590, 0x4004c10200, 0x200, 0x200, 0x4004c38640, 0x113402a, 0x40001f64f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000211640, 0x4004c10200, 0x200, 0x200, 0xc, 0x10, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000211640, 0x4004c10200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000211680, 0x4004c10200, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004c0e2c0, 0x0, 0x40001f66b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004c0e2c0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004c0e2c0, 0xee0d40, 0x400014ae10, 0x12db068, 0x4000211680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40046b59e0, 0x4004c6c060, 0x400469fb00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 31 [semacquire]: sync.runtime_Semacquire(0x4004e730c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004e730c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x40000b9b80, 0x400235c240, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffc4f18969, 0x29, 0xffffc4f189cb, 0x25, 0x4004e0b1a0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x40046d9aa0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffc4f18969, 0x29, 0xffffc4f189cb, 0x25, 0x4004e0b1a0, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x40046d9aa0, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x400457b190, 0x400457ae60, 0x400235c240, 0x400022b440, 0x40024681c0, 0x0, 0xbf, 0x40001c4da8, 0xffff7868fcb0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x4000182700, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x40046d9aa0, 0xb, 0x40046d9a98, 0x6, 0x40046d9ab0, 0xd, 0x40046ff320, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x4000182700, 0x40003e0000, 0x400011bbf0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x4000182700, 0x4000196280, 0x2, 0x2, 0x4000196290, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 162 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x4000188c40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 163 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x4000188c40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 164 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4004b6e200) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 177 [select]: net/http.(*persistConn).writeLoop(0x40044145a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 174 [select]: net/http.(*persistConn).readLoop(0x4004414480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 167 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004b70370, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 168 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x4004b70370, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 169 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x4004b70370) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 170 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x4000188c40) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 171 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x400006c710, 0x1144559, 0x1b, 0x4004d03e38, 0x4004649e00, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004677750, 0x4004649e00, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004d03f80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x400467ade0, 0x4004677750) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 178 [IO wait]: internal/poll.runtime_pollWait(0xffff7868a898, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000191598, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000191580, 0x4004e19000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000191580, 0x4004e19000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400013e8c8, 0x4004e19000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40044145a0, 0x4004e19000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x400467bb00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400467bb00, 0x100a, 0x0, 0x0, 0x0, 0x0, 0x40001f7438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x400467bb00, 0x0, 0x40001f7478, 0x655784, 0x40001f74a8, 0x40b2f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004771ad0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004771ad0, 0x400476dc00, 0x200, 0x200, 0x4, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004b6e380, 0x400476dc00, 0x200, 0x200, 0x4004774a87, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004b6e380, 0x400476dc00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004b6e3c0, 0x400476dc00, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004b72dc0, 0x0, 0x40001f76b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004b72dc0, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004b72dc0, 0xee0d40, 0x400008aff0, 0x12db068, 0x4004b6e3c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004e0c2d0, 0x400467bc80, 0x4004e0aba0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 176 [select]: net/http.(*persistConn).readLoop(0x40044145a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 175 [select]: net/http.(*persistConn).writeLoop(0x4004414480) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 179 [runnable]: net/http.(*persistConn).roundTrip(0x4004414480, 0x4004dd8140, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004bcba00, 0xf8, 0x10a8660, 0x4004d06e18) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004bcba00, 0x1b6e900, 0xc03eb95f8c1ae1df, 0x1bfa19ac22) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004bcb900, 0x12db048, 0x1b6e900, 0xc03eb95f8c1ae1df, 0x1bfa19ac22, 0x1be4a80, 0x0, 0x400235e2a8, 0x4004ddc330, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004ddc300, 0x4004bcb900, 0xc03eb95f8c1ae1df, 0x1bfa19ac22, 0x1be4a80, 0x400235e2a8, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004ddc300, 0x4004bcb900, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004d86870, 0x1b, 0x4004d07580, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004d61170, 0x112f905, 0x6, 0x12db908, 0x40001dd1e0, 0xee40c0, 0x4004499498, 0x4004d07580, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x4004b74200, 0x46, 0x12db908, 0x40001dd1e0, 0x0, 0x0, 0x400001b6a8, 0x412bc0, 0x40001dcd60, 0x20, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x4004b74200, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x20, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004d07d40, 0x4, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x40000d5d40, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x40000d5d40, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4004b6e240) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1267 +0xf8 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 180 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x4004b70210) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:224 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 181 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x4004b70210) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:612 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 182 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x4004b70210) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:662 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 183 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x4004b70210) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:752 +0x44 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 184 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004b55500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:249 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 197 [IO wait]: internal/poll.runtime_pollWait(0xffff7868a5e0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004499798, 0x72, 0x0, 0x1, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004499780, 0x4004ddc971, 0x1, 0x1, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004499780, 0x4004ddc971, 0x1, 0x1, 0x4004cf5f01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400235e2e0, 0x4004ddc971, 0x1, 0x1, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*connReader).backgroundRead(0x4004ddc960) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:692 +0x50 created by net/http.(*connReader).startBackgroundRead /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:688 +0xc4 goroutine 186 [select]: net/http.(*persistConn).roundTrip(0x400000cc60, 0x4004dd81c0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004bcbf00, 0xf8, 0x10a8660, 0x4004df0dd8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004bcbf00, 0x1b6e900, 0xc03eb95f8c1dd178, 0x1bfa1c9bc3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004bcbe00, 0x12db048, 0x1b6e900, 0xc03eb95f8c1dd178, 0x1bfa1c9bc3, 0x1be4a80, 0x0, 0x400235e2d0, 0x4004ddc660, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004ddc630, 0x4004bcbe00, 0xc03eb95f8c1dd178, 0x1bfa1c9bc3, 0x1be4a80, 0x400235e2d0, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004ddc630, 0x4004bcbe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004d86a80, 0x1b, 0x4004df1540, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004d61680, 0x112f905, 0x6, 0x12db908, 0x40001dd260, 0xee40c0, 0x4004499618, 0x4004df1540, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x4004bd4300, 0x46, 0x12db908, 0x40001dd260, 0x12f8668, 0x4004bcbc00, 0x4004b99e70, 0x4004d86990, 0xffff7893f410, 0x4004dd8180, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x4004bd4300, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x240, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4004df1d00, 0x4004ddc570, 0xffff7a9a9620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x40001ed200, 0xd47600, 0x230) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).FetchWithLock(0x40001ed200, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:384 +0x60 github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x4000182700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9702 +0x344 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 187 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x4000182700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1090 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 188 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x4000182700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1017 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 189 [sleep]: time.Sleep(0x12a05f200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/plasma.RunMemQuotaTunerIteration(0x4004e21fb0, 0xdf74d8, 0x400010e660) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:211 +0x24c github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:247 +0xc0 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60 goroutine 63 [select]: net/http.(*persistConn).roundTrip(0x40001eb7a0, 0x4004dd8180, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004bcbc00, 0xf8, 0x10a8660, 0x400001b798) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004bcbc00, 0x1b6e900, 0xc03eba6d8c1c6613, 0x1176f1f605e) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004bcbb00, 0x12db048, 0x1b6e900, 0xc03eba6d8c1c6613, 0x1176f1f605e, 0x1be4a80, 0x0, 0x400235e2b8, 0x4004ddc420, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004ddc3f0, 0x4004bcbb00, 0xc03eba6d8c1c6613, 0x1176f1f605e, 0x1be4a80, 0x400235e2b8, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004ddc3f0, 0x4004bcbb00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004dd69a0, 0x1f, 0x400001bd10, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/indexer.getWithAuth(0x4004dd69a0, 0x1f, 0xe, 0x113846e, 0x11) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:3077 +0x68 github.com/couchbase/indexing/secondary/indexer.getGlobalTopology(0x4004e726e0, 0xe, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:2266 +0x78 github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).updateNodeList(0x4004b55500) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:220 +0x3c created by github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:206 +0x98 goroutine 195 [IO wait]: internal/poll.runtime_pollWait(0xffff7868a7b0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004499598, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004499580, 0x4004f02000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004499580, 0x4004f02000, 0x1000, 0x1000, 0x40001f7be8, 0x7ff1a8, 0x40001f7cb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400235e2e8, 0x4004f02000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001eb7a0, 0x4004f02000, 0x1000, 0x1000, 0x60, 0x0, 0x4004dd6901) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004df8300) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004df8300, 0x1, 0x4004ddeb40, 0x43d584, 0x0, 0xd8c91c, 0x40001f7d68) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40001eb7a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 194 [select]: net/http.(*persistConn).roundTrip(0x40001eb9e0, 0x4004dd8440, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2610 +0x590 net/http.(*Transport).roundTrip(0x1b6e900, 0x4004dfc300, 0xf8, 0x10a8660, 0x4004d00588) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:592 +0x7e8 net/http.(*Transport).RoundTrip(0x1b6e900, 0x4004dfc300, 0x1b6e900, 0xc03eb95f8c21b8fb, 0x1bfa20833e) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/roundtrip.go:17 +0x30 net/http.send(0x4004dfc200, 0x12db048, 0x1b6e900, 0xc03eb95f8c21b8fb, 0x1bfa20833e, 0x1be4a80, 0x0, 0x400235e2f8, 0x4004ddcc60, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:251 +0x348 net/http.(*Client).send(0x4004ddcc30, 0x4004dfc200, 0xc03eb95f8c21b8fb, 0x1bfa20833e, 0x1be4a80, 0x400235e2f8, 0x0, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:175 +0xc8 net/http.(*Client).do(0x4004ddcc30, 0x4004dfc200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:717 +0x33c net/http.(*Client).Do(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/client.go:585 github.com/couchbase/indexing/secondary/security.GetWithAuthAndETag(0x4004d86d50, 0x1b, 0x4004d00cf0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:551 +0x1a4 github.com/couchbase/indexing/secondary/security.GetWithAuth(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/security/tls.go:504 github.com/couchbase/indexing/secondary/dcp.queryRestAPI(0x4004d61cb0, 0x112f905, 0x6, 0x12db908, 0x40001dd2c0, 0xee40c0, 0x4004499898, 0x4004d00cf0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:337 +0x134 github.com/couchbase/indexing/secondary/dcp.(*Client).parseURLResponse(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:470 github.com/couchbase/indexing/secondary/dcp.ConnectWithAuth(0x4004bd4380, 0x46, 0x12db908, 0x40001dd2c0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:532 +0x108 github.com/couchbase/indexing/secondary/dcp.Connect(0x4004bd4380, 0x46, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:538 +0x94 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch.func1(0x0, 0x0, 0x0, 0x4004ddcb70, 0xffff7a9a9620) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:289 +0x37c github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400457f4b0, 0x1063520, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/common.(*ClusterInfoCache).Fetch(0x40001ed440, 0x40001ed440, 0x4004bd4380) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:377 +0x70 github.com/couchbase/indexing/secondary/common.FetchNewClusterInfoCache(0xffffc4f1894e, 0xe, 0x113075a, 0x7, 0x1130e44, 0x8, 0x4004f000c4, 0x2, 0xf01) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:123 +0x134 github.com/couchbase/indexing/secondary/manager.(*IndexManager).FetchNewClusterInfoCache(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:306 github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).getIndexMetadata(0x1be5520, 0x12ec7a0, 0x4004dd8400, 0x40001f31d0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:1284 +0x74 github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).handleIndexMetadataRequest(0x1be5520, 0x12edbe0, 0x400042efc0, 0x4004dfc100) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:1271 +0x3e8 net/http.HandlerFunc.ServeHTTP(0x40045385c0, 0x12edbe0, 0x400042efc0, 0x4004dfc100) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2069 +0x40 net/http.(*ServeMux).ServeHTTP(0x4002469500, 0x12edbe0, 0x400042efc0, 0x4004dfc100) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2448 +0x190 net/http.serverHandler.ServeHTTP(0x400253a540, 0x12edbe0, 0x400042efc0, 0x4004dfc100) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2887 +0xbc net/http.(*conn).serve(0x4004d8a5a0, 0x12f86a0, 0x4004dd8240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:1952 +0x71c created by net/http.(*Server).Serve /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:3013 +0x318 goroutine 196 [select]: net/http.(*persistConn).writeLoop(0x40001eb7a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 200 [IO wait]: internal/poll.runtime_pollWait(0xffff7868a4f8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004499998, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004499980, 0x4004f04000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004499980, 0x4004f04000, 0x1000, 0x1000, 0x40001f7be8, 0x7ff1a8, 0x40001f7cb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400235e308, 0x4004f04000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001eb9e0, 0x4004f04000, 0x1000, 0x1000, 0x60, 0x0, 0x4004dd6901) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004df8840) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4004df8840, 0x1, 0x4004ddef60, 0x43d584, 0x0, 0xd8c91c, 0x40001f7d68) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40001eb9e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 201 [select]: net/http.(*persistConn).writeLoop(0x40001eb9e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc Initializing write barrier = 8000 2021-08-16T13:52:11.372+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 --httpsPort=19102 --certFile=/opt/couchbase/var/lib/couchbase/config/memcached-cert.pem --keyFile=/opt/couchbase/var/lib/couchbase/config/memcached-key.pem -ipv4=required -ipv6=optional -vbuckets=1024 -cluster=127.0.0.1:8091 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -logDir=/opt/couchbase/var/lib/couchbase/logs -nodeUUID=8a95cde683482bdec453edbb8b18eb5d -isEnterprise=true] 2021-08-16T13:52:11.372+00:00 [Info] Setting ipv6=false 2021-08-16T13:52:11.381+00:00 [Info] Indexer::NewIndexer Status Warmup 2021-08-16T13:52:11.383+00:00 [Info] Setting buffer block size to 16384 bytes 2021-08-16T13:52:11.383+00:00 [Info] Setting maxcpus = 4 2021-08-16T13:52:11.383+00:00 [Info] Setting log level to Info 2021-08-16T13:52:11.383+00:00 [Info] audit::InitAuditService using address http://127.0.0.1:8091 2021-08-16T13:52:11.386+00:00 [Info] SettingsMgr::metaKVCallback Dropped request /indexing/settings/config {"indexer.settings.rebalance.redistribute_indexes":false,"indexer.settings.compaction.days_of_week":"Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday","indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.compaction_mode":"circular","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":0,"indexer.settings.storage_mode":"plasma","indexer.settings.recovery.max_rollbacks":2,"indexer.settings.memory_quota":536870912,"indexer.settings.num_replica":0,"indexer.settings.compaction.abort_exceed_interval":false}. Any setting change will get applied once Indexer is ready. 2021/08/16 13:52:11 audit: created new audit service 2021-08-16T13:52:11.394+00:00 [Info] Recieve security change notification. code 3 2021-08-16T13:52:11.394+00:00 [Info] TLS config refreshed successfully 2021-08-16T13:52:11.395+00:00 [Info] Certificate refreshed successfully 2021-08-16T13:52:11.395+00:00 [Info] Encryption config refresh successfully. Encryption enabled=false 2021-08-16T13:52:11.395+00:00 [Info] security context: encryptLocalHost true Local IP's: map[127.0.0.1:true 172.31.29.9:true ::1:true fe80::80f:d6ff:fe0f:764a:true] 2021-08-16T13:52:11.405+00:00 [Info] security context initialized 2021-08-16T13:52:11.405+00:00 [Info] security port mapping updated : map[11210:11207 8093:18093 9100:9100 9101:9101 9102:19102 9103:9103 9104:9104 9105:9105 9999:9999] 2021-08-16T13:52:11.405+00:00 [Info] Indexer::NewIndexer Build Mode Set Enterprise 2021-08-16T13:52:11.405+00:00 [Info] Indexer::Cluster Storage Mode Set plasma 2021-08-16T13:52:11.405+00:00 [Info] memcachedTimeout set to 120 2021-08-16T13:52:11.405+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2021-08-16T13:52:11.417+00:00 [Info] MutationMgr::checkPortAvailability for dataport servers ... 2021-08-16T13:52:11.418+00:00 [Info] MutationMgr::checkPortAvailability all dataport server ports are available. 2021-08-16T13:52:11.437+00:00 [Info] WatchClusterVersionChanges: Cluster version is >= INDEXER_70_VERSION 2021-08-16T13:52:11.447+00:00 [Info] [Queryport ":9101"] started ... 2021-08-16T13:52:11.448+00:00 [Info] OperationsMonitor(kvSender) starting monitor ... 2021-08-16T13:52:11.448+00:00 [Info] serviceChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default 2021-08-16T13:52:11.449+00:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap 2021-08-16T13:52:11.449+00:00 [Info] serviceChangeNotifier: received ServiceChangeNotification 2021-08-16T13:52:11.450+00:00 [Info] serviceChangeNotifier: Starting to monitor the bucket streaming endpoint for bucket: travel-sample 2021-08-16T13:52:11.450+00:00 [Info] serviceChangeNotifier: received PoolChangeNotification 2021-08-16T13:52:11.451+00:00 [Info] bootstrap storage mode plasma 2021-08-16T13:52:11.496+00:00 [Info] Starting metadadta repo: quota 1048576 sleep duration 900 threshold 30 min file size 0 2021-08-16T13:52:11.497+00:00 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 54 us 2021-08-16T13:52:11.497+00:00 [INFO][FDB] Forestdb opened database file /opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore.0 2021-08-16T13:52:11.500+00:00 [Info] Repo.upgradeAndOpenDBFile(/opt/couchbase/var/lib/couchbase/data/@2i/MetadataStore): Opened with COMPACT_AUTO mode 2021-08-16T13:52:11.500+00:00 [Info] EmbeddedServer.runOnce() : Start Running Server 2021-08-16T13:52:11.504+00:00 [Info] indexer:: Staring http server : :9102 2021-08-16T13:52:11.504+00:00 [Info] HTTPS server created for :19102 2021-08-16T13:52:11.504+00:00 [Info] indexer:: SSL server started: :19102 2021-08-16T13:52:11.504+00:00 [Info] Indexer::indexer version 5 2021-08-16T13:52:11.504+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2021-08-16T13:52:11.504+00:00 [Info] Indexer Id 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:52:11.504+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:52:11.505+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic MAINT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:52:11.505+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:52:11.505+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:52:11.506+00:00 [Info] KVSender::sendShutdownTopic Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:52:11.506+00:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 Topic INIT_STREAM_TOPIC_8a95cde683482bdec453edbb8b18eb5d. Err projector.topicMissing 2021-08-16T13:52:11.506+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from ec2-3-8-56-217.eu-west-2.compute.amazonaws.com:9999 2021-08-16T13:52:11.506+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2021-08-16T13:52:11.506+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning 2021-08-16T13:52:11.506+00:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken 2021-08-16T13:52:11.506+00:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken 2021-08-16T13:52:11.506+00:00 [Info] ClustMgr:handleGetGlobalTopology indexInstMap : 2021-08-16T13:52:11.512+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap InstanceId: 633044510811627291 Name: def_primary Keyspace: travel-sample:_default:_default State: INDEX_STATE_READY Stream: NIL_STREAM RState: RebalActive Version: 0 ReplicaId: 0 , elapsed: 5.873532ms 2021-08-16T13:52:11.512+00:00 [Info] ClustMgr:handleUpdateTopologyForIndex mType: CLUST_MGR_UPDATE_TOPOLOGY_FOR_INDEX indexList: [] bucket: scope: collection: streamId: NIL_STREAM syncUpdate: false respCh: 2021-08-16T13:52:11.513+00:00 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set 2021-08-16T13:52:11.513+00:00 [Info] DDLServiceMgr: intialized. Local nodeUUID 8a95cde683482bdec453edbb8b18eb5d 2021-08-16T13:52:11.514+00:00 [Info] Indexer::local storage mode plasma 2021-08-16T13:52:11.514+00:00 [Info] DDLServiceMgr: starting buildTokenCleaner ... 2021-08-16T13:52:11.514+00:00 [Info] DDLServiceMgr: starting delTokenCleaner ... 2021-08-16T13:52:11.514+00:00 [Info] DDLServiceMgr: starting dropInstTokenCleaner ... 2021-08-16T13:52:11.525+00:00 [Info] schedIndexCreator: intialized. 2021-08-16T13:52:11.525+00:00 [Info] RebalanceMgr::NewRebalanceMgr false 2021-08-16T13:52:11.525+00:00 [Info] Indexer::initPartnInstance Initialized Partition: Index: 633044510811627291 Partition: PartitionId: 0 Endpoints: [:9105] 2021-08-16T13:52:11.525+00:00 [Info] Start shard recovery from /opt/couchbase/var/lib/couchbase/data/@2i/shards 2021-08-16T13:52:11.526+00:00 [Info] : Recover shard shard1 from metadata completed. unexpected fault address 0x4004bdafdf fatal error: fault [signal SIGBUS: bus error code=0x1 addr=0x4004bdafdf pc=0xbd8ea4] goroutine 186 [running]: runtime.throw(0x112f565, 0x5) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/panic.go:1117 +0x54 fp=0x4004dce9f0 sp=0x4004dce9c0 pc=0x43ab14 runtime.sigpanic() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/signal_unix.go:731 +0x284 fp=0x4004dcea30 sp=0x4004dce9f0 pc=0x452254 github.com/couchbase/nitro/skiplist.(*Node).getNext(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/node_multi_arch.go:110 github.com/couchbase/nitro/skiplist.(*Skiplist).findPath(0x4004d364e0, 0x400475d800, 0x11a9890, 0x4004dceb40, 0x4004d364f8, 0xc2fa0c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:218 +0x274 fp=0x4004dceab0 sp=0x4004dcea40 pc=0xbd8ea4 github.com/couchbase/nitro/skiplist.(*Skiplist).Lookup(0x4004d364e0, 0x400475d800, 0x11a9890, 0x4004dceb40, 0x4004d364f8, 0xc18f8c, 0x4004dcec18, 0x4dd5a4) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/nitro/skiplist/skiplist.go:204 +0x48 fp=0x4004dceaf0 sp=0x4004dceab0 pc=0xbd8bb8 github.com/couchbase/plasma.(*Shard).FindInstance(0x4000134f00, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1659 +0x114 fp=0x4004dceda0 sp=0x4004dceaf0 pc=0xc2faa4 github.com/couchbase/plasma.(*Shard).AssignPlasmaId(0x4000134f00, 0x4004eb8150, 0x68, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:1188 +0xd8 fp=0x4004dcee60 sp=0x4004dceda0 pc=0xc2bfb8 github.com/couchbase/plasma.initShard(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004d32b90, 0x4004d32ba0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:379 +0xb0 fp=0x4004dcf200 sp=0x4004dcee60 pc=0xc10d00 github.com/couchbase/plasma.New3(0x0, 0xc8, 0x190, 0x19, 0x4, 0x11a98c0, 0x11a9898, 0x11a98d0, 0x4004d32b90, 0x4004d32ba0, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/plasma.go:320 +0x108 fp=0x4004dcf8f0 sp=0x4004dcf200 pc=0xc107e8 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores.func1(0x4004d53b50, 0x4004d7ca80, 0x400015a680, 0x4004d32b70) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:411 +0xb0 fp=0x4004dcffb0 sp=0x4004dcf8f0 pc=0xe4cd40 runtime.goexit() /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/asm_arm64.s:1130 +0x4 fp=0x4004dcffb0 sp=0x4004dcffb0 pc=0x4714f4 created by github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:408 +0x2da0 goroutine 1 [select]: github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0x400014c000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:969 +0x380 github.com/couchbase/indexing/secondary/indexer.NewIndexer(0x400010f680, 0x4, 0x1136a87, 0xf, 0x40000f9e68) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:461 +0x195c main.main() goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:180 +0x1128 goroutine 18 [select]: github.com/couchbase/cbauth/cbauthimpl.(*tlsNotifier).loop(0x4000122108) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:352 +0x64 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:505 +0x238 goroutine 19 [select]: github.com/couchbase/cbauth/cbauthimpl.(*cfgChangeNotifier).loop(0x4000122120) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:272 +0x74 created by github.com/couchbase/cbauth/cbauthimpl.NewSVCForTest /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/cbauthimpl/impl.go:506 +0x25c goroutine 20 [IO wait]: internal/poll.runtime_pollWait(0xffff61353758, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0180, 0x4000301000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0180, 0x4000301000, 0x1000, 0x1000, 0x1311860, 0x7b, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010500, 0x4000301000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 bufio.(*Reader).Read(0x4000082480, 0x4000306001, 0xdff, 0xdff, 0x400009f8e8, 0x6902d8, 0x40000f5948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:227 +0x214 github.com/couchbase/cbauth/revrpc.(*minirwc).Read(0x400000e5b8, 0x4000306001, 0xdff, 0xdff, 0x191, 0x1, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:101 +0x44 encoding/json.(*Decoder).refill(0x40000ceb00, 0xfe220a, 0x9) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x40000ceb00, 0x0, 0x0, 0x68f120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x40000ceb00, 0xf5a960, 0x4000082560, 0x6f0d60, 0xfe22a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 net/rpc/jsonrpc.(*serverCodec).ReadRequestHeader(0x4000082540, 0x4000046340, 0x40000f5b28, 0x49fb78) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/jsonrpc/server.go:66 +0x64 net/rpc.(*Server).readRequestHeader(0x400008c190, 0x12f8f98, 0x4000082540, 0x4000036cd7, 0x16, 0x400000d200, 0x400009fc01, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:583 +0x4c net/rpc.(*Server).readRequest(0x400008c190, 0x12f8f98, 0x4000082540, 0x400008c190, 0x40000364e8, 0x40000364f0, 0x40000f0200, 0x4000046260, 0xee0980, 0x400000d200, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:543 +0x38 net/rpc.(*Server).ServeCodec(0x400008c190, 0x12f8f98, 0x4000082540) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/rpc/server.go:458 +0x70 github.com/couchbase/cbauth/revrpc.(*Service).Run(0x400010e600, 0x400009ff78, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:147 +0x49c github.com/couchbase/cbauth/revrpc.BabysitService(0x400009df78, 0x400010e600, 0x12db8a8, 0x400000e588, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/revrpc/revrpc.go:243 +0x50 github.com/couchbase/cbauth.runRPCForSvc(0x400010e600, 0x4000132460, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:57 +0xc0 github.com/couchbase/cbauth.startDefault.func1(0x400010e600, 0x4000132460) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:66 +0x30 created by github.com/couchbase/cbauth.startDefault /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/default.go:65 +0xdc goroutine 39 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400018c780) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 21 [sleep]: time.Sleep(0x2540be400) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.pollForDeletedBuckets() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:1062 +0x30 created by github.com/couchbase/indexing/secondary/common.init.1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/dcp_seqno.go:52 +0xac goroutine 22 [chan receive]: github.com/couchbase/plasma.runCleanerAutoTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:651 +0x16c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:183 +0x124 goroutine 23 [chan receive]: github.com/couchbase/plasma.singletonWorker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2340 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:184 +0x138 goroutine 24 [chan receive]: github.com/couchbase/plasma.systemResourceTracker() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:276 +0xa0 created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:185 +0x14c goroutine 25 [chan receive]: github.com/couchbase/plasma.AggregateAndLogStats() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:2308 +0x8c created by github.com/couchbase/plasma.init.2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/shard.go:186 +0x160 goroutine 34 [chan receive]: github.com/couchbase/plasma.(*CleanerAutoTuner).refreshCleanerBandwidth(0x4000194000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:662 +0x7c created by github.com/couchbase/plasma.runCleanerAutoTuner /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/auto_tuner.go:648 +0x104 goroutine 26 [syscall]: syscall.Syscall(0x3f, 0x0, 0x4000055fa0, 0x4, 0x4000055e68, 0x4cc6d0, 0x800000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/asm_linux_arm64.s:9 +0x10 syscall.read(0x0, 0x4000055fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/zsyscall_linux_arm64.go:686 +0x4c syscall.Read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/syscall/syscall_unix.go:187 internal/poll.ignoringEINTRIO(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:581 internal/poll.(*FD).Read(0x4000082060, 0x4000055fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:162 +0x100 os.(*File).read(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file_posix.go:31 os.(*File).Read(0x4000010010, 0x4000055fa0, 0x4, 0x4, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/os/file.go:117 +0x60 github.com/couchbase/indexing/secondary/common.ExitOnStdinClose() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/util.go:407 +0x48 created by main.main goproj/src/github.com/couchbase/indexing/secondary/cmd/indexer/main.go:87 +0xa60 goroutine 37 [select]: net/http.(*persistConn).readLoop(0x400000c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 38 [select]: net/http.(*persistConn).writeLoop(0x400000c5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 10 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000112750, 0x12c18bc, 0x1, 0x400034de58, 0x4000144180, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x12c18bc, 0x1, 0x400019eec8, 0x4000144180, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:77 +0xf4 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400034dfa0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/indexer.NewSettingsManager.func1(0x4000114000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:81 +0x74 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:72 +0x178 goroutine 11 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*settingsManager).run(0x4000114000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:252 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewSettingsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/settings.go:88 +0x198 goroutine 16 [IO wait]: internal/poll.runtime_pollWait(0xffff61353670, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000124198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000124180, 0x4000307000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000124180, 0x4000307000, 0x1000, 0x1000, 0x4000065a88, 0x68f7a4, 0x400012ecb0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000186018, 0x4000307000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000c5a0, 0x4000307000, 0x1000, 0x1000, 0xf12180, 0x4000128310, 0x181) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000180660) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000180660, 0xa, 0x1008880, 0x40001282d0, 0x199, 0xad9, 0x4000065b00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000180660, 0x2, 0x0, 0x9, 0x56b, 0xb) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4000443140) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4000443140, 0x4000354000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000114300, 0x4000354000, 0xe00, 0xe00, 0x744, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000114300, 0x4000354000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000114340, 0x4000354000, 0xe00, 0xe00, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x400012ec60, 0x0, 0x4000065eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400012ec60, 0x0, 0x0, 0x4000065f70) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400012ec60, 0xee0d40, 0x40001282d0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40001477a0, 0x40003f7620, 0x4000144a20) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 14 [IO wait]: internal/poll.runtime_pollWait(0xffff61353588, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000124298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000124280, 0x4000350000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000124280, 0x4000350000, 0x1000, 0x1000, 0x4000346be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011a090, 0x4000350000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400000cc60, 0x4000350000, 0x1000, 0x1000, 0x4000346cf8, 0x67867c, 0x40001449c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40003f74a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40003f74a0, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400000cc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 15 [select]: net/http.(*persistConn).writeLoop(0x400000cc60) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 116 [IO wait]: internal/poll.runtime_pollWait(0xffff61352aa8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000124518, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000124500, 0x40024a9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000124500, 0x40024a9000, 0x1000, 0x1000, 0x4000343be0, 0x469b10, 0x400141ed80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010668, 0x40024a9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40000f47e0, 0x40024a9000, 0x1000, 0x1000, 0x4000343cf8, 0x67867c, 0x40000395c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40024b8000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40024b8000, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40000f47e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 120 [select]: net/http.(*persistConn).readLoop(0x40000f4240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 122 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40001150c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 119 [select]: net/http.(*persistConn).writeLoop(0x4001452120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 41 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run(0x40000e87e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:162 +0xdc created by github.com/couchbase/indexing/secondary/indexer.NewMutationManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:141 +0x3d0 goroutine 71 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*kvSender).run(0x4000443800) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:102 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewKVSender /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/kv_sender.go:87 +0x1f0 goroutine 70 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/common.(*OperationsMonitor).monitor(0x40003f7c80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:199 +0x1a0 created by github.com/couchbase/indexing/secondary/common.NewOperationsMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/op_monitor.go:98 +0xf8 goroutine 44 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).handleWorkerMsgs(0x40000e87e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:186 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:154 +0x60 goroutine 45 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).listenWorkerMsgs(0x40000e87e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:209 +0x74 created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:155 +0x80 goroutine 46 [select]: github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).processStreamBegins(0x40000e87e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:391 +0x6c created by github.com/couchbase/indexing/secondary/indexer.(*mutationMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/mutation_manager.go:156 +0xa0 goroutine 69 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x400018cb80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 49 [select]: net/http.(*persistConn).readLoop(0x40001779e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 66 [select]: net/http.(*persistConn).writeLoop(0x40001779e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 72 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*timekeeper).run(0x40001b2680) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:115 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewTimekeeper /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/timekeeper.go:100 +0x124 goroutine 73 [IO wait]: internal/poll.runtime_pollWait(0xffff61352f30, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001b2898, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40001b2880, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40001b2880, 0x0, 0x4000058e98, 0xccadf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x400241dfe0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x400241dfe0, 0x12ed9d0, 0x400241dfe0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/indexing/secondary/queryport.(*Server).listener(0x40001b2700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:222 +0xa8 created by github.com/couchbase/indexing/secondary/queryport.NewServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/queryport/server.go:91 +0x4f4 goroutine 74 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40001275f0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 75 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).listenSnapshot(0x40001275f0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:152 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:117 +0x39c goroutine 76 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*scanCoordinator).run(0x40001275f0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:131 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewScanCoordinator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/scan_coordinator.go:121 +0x3d0 goroutine 77 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*compactionManager).run(0x4000097b60) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:704 +0x54 created by github.com/couchbase/indexing/secondary/indexer.NewCompactionManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:694 +0x8c goroutine 83 [IO wait]: internal/poll.runtime_pollWait(0xffff61352c78, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40023ea418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40023ea400, 0x40023ee000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40023ea400, 0x40023ee000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40001860f0, 0x40023ee000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4000176a20, 0x40023ee000, 0x1000, 0x1000, 0x400141e900, 0x200000003, 0x400141e900) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40023e83c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40023e83c0, 0x40001cc00a, 0x400034f8f8, 0x412bc0, 0x40000820c0, 0x40001cc000, 0x5c) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40023e83c0, 0x0, 0x1, 0x0, 0x40000820d8, 0x40001cc000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400030ccf0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400030ccf0, 0x4001462000, 0x1000, 0x1000, 0x400034f9b8, 0x44d8a8, 0x40) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000081880, 0x4001462000, 0x1000, 0x1000, 0x4000082bf8, 0x4000081940, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000081880, 0x4001462000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x40000818c0, 0x4001462000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400034fdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400034fdd0, 0xa, 0x4001462201, 0x1, 0xd1e, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400034fdd0, 0x400146220a, 0x1, 0x400047f069, 0x0, 0x40014622e2, 0x1, 0xd1e, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400034fdd0, 0x4000099b0a, 0x400047f069, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000127700, 0x400003a918, 0x17, 0x400034ff20, 0x40023e4978, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObservePool(0x4000127700, 0x113075a, 0x7, 0x40023e4978, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:371 +0x98 github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunPoolObserver(0x40001276c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:140 +0x5c created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:243 +0x5a4 goroutine 94 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run(0x40000e90a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:183 +0x78 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:145 +0x9d4 goroutine 80 [select]: github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).loop(0x40003d4700) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:122 +0x120 created by github.com/couchbase/indexing/secondary/indexer.(*compactionDaemon).Start /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/compaction_manager.go:70 +0x110 goroutine 89 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x40004777c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 84 [IO wait]: internal/poll.runtime_pollWait(0xffff61352e48, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0b98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0b80, 0x4002449000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0b80, 0x4002449000, 0x1000, 0x1000, 0x40, 0x1d, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000186260, 0x4002449000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40001779e0, 0x4002449000, 0x1000, 0x1000, 0x400141ef00, 0x200000003, 0x400141ef00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000181f20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4000181f20, 0x40001cc00a, 0x400009f8c8, 0x4d3b10, 0x40000820c0, 0x40001cc000, 0x5f) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4000181f20, 0x0, 0x1, 0x1c211e8, 0x40000820d8, 0x40001cc000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x400030c6c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x400030c6c0, 0x400144d000, 0x1000, 0x1000, 0x400009f9b8, 0x0, 0x400009f988) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x40000817c0, 0x400144d000, 0x1000, 0x1000, 0x4000001b00, 0x3, 0x400009fbe0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x40000817c0, 0x400144d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000081800, 0x400144d000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400009fdd0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400009fdd0, 0xa, 0x400144d201, 0x1, 0xd9d, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400009fdd0, 0x400144d20a, 0x1, 0x400047edfb, 0x0, 0x400144d263, 0x1, 0xd9d, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400009fdd0, 0x400143200a, 0x400047edfb, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000127700, 0x400011cc00, 0x24, 0x400009ff20, 0x40024501f8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveNodeServices(0x4000127700, 0x113075a, 0x7, 0x40024501f8, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:387 +0xac github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunServicesObserver(0x40001276c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:149 +0x58 created by github.com/couchbase/indexing/secondary/common.NewServicesChangeNotifier /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:244 +0x5c4 goroutine 100 [select]: net/http.(*persistConn).readLoop(0x4000176a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 101 [select]: net/http.(*persistConn).writeLoop(0x4000176a20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 121 [select]: net/http.(*persistConn).writeLoop(0x40000f4240) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 118 [select]: net/http.(*persistConn).readLoop(0x4001452120) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 117 [select]: net/http.(*persistConn).writeLoop(0x40000f47e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 53 [IO wait]: internal/poll.runtime_pollWait(0xffff61353018, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000124418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4000124400, 0x40024bf000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4000124400, 0x40024bf000, 0x1000, 0x1000, 0x400246b688, 0x69f178, 0x400246b6f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010680, 0x40024bf000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40000f4240, 0x40024bf000, 0x1000, 0x1000, 0x5, 0x4002542500, 0x400246b7c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40024b8180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40024b8180, 0x68f90a, 0x4002512a50, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40024b8180, 0x0, 0x2, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4002511170) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4002511170, 0x40024df000, 0x1000, 0x1000, 0x400246b998, 0x44e884, 0x400246b948) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4000476840, 0x40024df000, 0x1000, 0x1000, 0x400244e718, 0x44e778, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4000476840, 0x40024df000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4000476880, 0x40024df000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 bufio.(*Reader).fill(0x400246bd80) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x400246bd80, 0xa, 0x40024df201, 0x1, 0xffff607349a8, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 bufio.(*Reader).collectFragments(0x400246bd80, 0x40024df20a, 0x1, 0x400469c0b0, 0x0, 0x40024df259, 0x1, 0xda7, 0x1, 0x0, ...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:435 +0x64 bufio.(*Reader).ReadBytes(0x400246bd80, 0x40024b690a, 0x400469c0b0, 0x1, 0x1, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:463 +0x30 github.com/couchbase/indexing/secondary/dcp.(*Client).runObserveStreamingEndpoint(0x4000127700, 0x400145c200, 0x1f, 0x400246bee0, 0x40001f68b8, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:443 +0x378 github.com/couchbase/indexing/secondary/dcp.(*Client).RunObserveCollectionManifestChanges(0x4000127700, 0x113075a, 0x7, 0x400047f080, 0xd, 0x40001f68b8, 0x0, 0x40024a0b80, 0x2) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/dcp/pools.go:402 +0xbc github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).RunObserveCollectionManifestChanges(0x40001276c0, 0x400047f080, 0xd) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:158 +0x6c created by github.com/couchbase/indexing/secondary/common.(*serviceNotifierInstance).bucketsChangeCallback /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/services_notifier.go:130 +0x2ec goroutine 93 [semacquire]: sync.runtime_Semacquire(0x4004d53b58) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/sema.go:56 +0x38 sync.(*WaitGroup).Wait(0x4004d53b50) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/sync/waitgroup.go:130 +0x74 github.com/couchbase/indexing/secondary/indexer.(*plasmaSlice).initStores(0x400015a680, 0x4000442c90, 0x11573ee) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:431 +0x2db8 github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice(0xffffcc152969, 0x29, 0xffffcc1529cb, 0x25, 0x4004d50900, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004c6af10, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:229 +0xaa8 github.com/couchbase/indexing/secondary/indexer.NewPlasmaSlice(0xffffcc152969, 0x29, 0xffffcc1529cb, 0x25, 0x4004d50900, 0x5e, 0x0, 0xb0f3756ee93d3573, 0x4004c6af10, 0xb, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_enterprise.go:25 +0xbc github.com/couchbase/indexing/secondary/indexer.NewSlice(0x0, 0x4004dcf190, 0x4004dcee60, 0x4000442c90, 0x40000b9b00, 0x400018c780, 0x0, 0xffff60749ee0, 0x98, 0xffff8ba26a90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:8375 +0x5ac github.com/couchbase/indexing/secondary/indexer.(*indexer).initPartnInstance(0x400014c000, 0x8c906a0e9f01f1b, 0xb0f3756ee93d3573, 0x4004c6af10, 0xb, 0x4004c6af08, 0x6, 0x4004c6af20, 0xd, 0x4004c67700, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4821 +0x1e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).initFromPersistedState(0x400014c000, 0x40003f0000, 0x40001822a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:7067 +0x3e0 github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1.func1(0x400014c000, 0x40001ee790, 0x2, 0x2, 0x40001ee7a0, 0x2, 0x2, 0x40bd00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6569 +0x258 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).bootstrap1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:6556 +0x448 goroutine 123 [select]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4000476d80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1287 +0x328 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 90 [IO wait]: internal/poll.runtime_pollWait(0xffff613529c0, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001b3d98, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x40001b3d80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x40001b3d80, 0x58ec48, 0x4000145bc0, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004c65fe0, 0x4714f4, 0x4000145bc0, 0x12c2e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004c65fe0, 0x4000037b00, 0x0, 0x4714f8, 0x11ae310) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c github.com/couchbase/gometa/common.(*PeerListener).listen(0x4004800980) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:220 +0x84 created by github.com/couchbase/gometa/common.startPeerListener /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/common/peerListener.go:74 +0xf0 goroutine 91 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).processRequest(0x4000477e80, 0x400014b5e0, 0x400014b650, 0x400469b9c0, 0x12e8f90, 0x40025547e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:386 +0x1bc github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler(0x4004c6a224, 0x5, 0x4004800980, 0x12eca70, 0x4004800840, 0x130d478, 0x40003d47e0, 0x1309878, 0x1c1c218, 0x12e8f90, ...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:131 +0x3f0 github.com/couchbase/gometa/server.(*EmbeddedServer).runOnce(0x40000e8fc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:678 +0x11c github.com/couchbase/gometa/server.(*EmbeddedServer).run(0x40000e8fc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:578 +0x54 created by github.com/couchbase/gometa/server.RunEmbeddedServerWithCustomHandler2 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/server/embeddedServer.go:106 +0x1d0 goroutine 124 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).listen(0x4000189540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:556 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:180 +0x23c goroutine 125 [chan receive]: github.com/couchbase/gometa/protocol.(*Leader).processResponses(0x4000189540) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:601 +0xa0 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:181 +0x25c goroutine 126 [select]: github.com/couchbase/gometa/protocol.(*LeaderServer).listenFollower(0x4000477e80, 0x400469b9c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:187 +0xdc created by github.com/couchbase/gometa/protocol.RunLeaderServerWithCustomHandler /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/gometa/protocol/leaderServer.go:126 +0x3bc goroutine 127 [select]: github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).processRequest(0x40025547e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:328 +0x114 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:202 +0x78 goroutine 128 [sleep]: time.Sleep(0x3b9aca00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).broadcastStats(0x40025547e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:2645 +0x54 created by github.com/couchbase/indexing/secondary/manager.(*LifecycleMgr).Run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/lifecycle.go:203 +0x98 goroutine 129 [select]: github.com/couchbase/indexing/secondary/manager.(*IndexManager).monitorKeyspace(0x400243aa20, 0x40047beba0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:700 +0xa4 created by github.com/couchbase/indexing/secondary/manager.NewIndexManagerInternal /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/manager.go:212 +0x890 goroutine 130 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*clustMgrAgent).run(0x4000115080) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:109 +0xb0 created by github.com/couchbase/indexing/secondary/indexer.NewClustMgrAgent /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cluster_manager_agent.go:82 +0x298 goroutine 131 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).run(0x40001895e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3011 +0x38 created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2571 +0x36c goroutine 132 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*statsManager).runStatsDumpLogger(0x40001895e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3175 +0x8c created by github.com/couchbase/indexing/secondary/indexer.NewStatsManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:2572 +0x38c goroutine 133 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*cpuCollector).runCollectStats(0x4000010b28) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:70 +0x94 created by github.com/couchbase/indexing/secondary/indexer.StartCpuCollector /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/cpu.go:54 +0x138 goroutine 134 [select]: github.com/couchbase/indexing/secondary/indexer.(*statsManager).statsUpdater(0x40001895e0, 0x40047bec00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3066 +0x13c created by github.com/couchbase/indexing/secondary/indexer.(*statsManager).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/stats_manager.go:3029 +0xf8 goroutine 135 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000112750, 0x1144559, 0x1b, 0x4002469e38, 0x40047bed20, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004d8a4e0, 0x40047bed20, 0xf1f700, 0x4000113fd0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x1ba2168, 0x1177289) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x4002469f80, 0x116fb9d, 0x3c) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x40047f0d20, 0x4004d8a4e0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 136 [select]: github.com/couchbase/indexing/secondary/manager.(*requestHandlerContext).runPersistor(0x1be5520) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:2945 +0xf8 created by github.com/couchbase/indexing/secondary/manager.RegisterRequestHandler.func1 /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/request_handler.go:296 +0xbd4 goroutine 137 [chan receive]: github.com/couchbase/indexing/secondary/logging.PeriodicProfile.func1(0x400008d130, 0xd00007, 0x4004d80578, 0x5, 0x4004d8a4f0, 0x1, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:207 +0x30 created by github.com/couchbase/indexing/secondary/logging.PeriodicProfile /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:204 +0x80 goroutine 138 [IO wait]: internal/poll.runtime_pollWait(0xffff613528d8, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000125d18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000125d00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000125d00, 0xffff8ba265b8, 0x30, 0x4004d84ae0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004d86288, 0x55d18c, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004d86288, 0x4004d84ae0, 0x11ae500, 0x4002554be0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c net/http.(*Server).Serve(0x4002554b60, 0x12ed9d0, 0x4004d86288, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer.func1(0x4002554b60, 0x12ed9d0, 0x4004d86288, 0x400014c000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:710 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:708 +0x450 goroutine 139 [IO wait]: internal/poll.runtime_pollWait(0xffff60747598, 0x72, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4000125e18, 0x72, 0x0, 0x0, 0x1130649) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0x4000125e00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:401 +0x1b0 net.(*netFD).accept(0x4000125e00, 0x30, 0x30, 0x101b5a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_unix.go:172 +0x2c net.(*TCPListener).accept(0x4004d862a0, 0xffff8ba265b8, 0xffff6043b7e8, 0x30) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock_posix.go:139 +0x2c net.(*TCPListener).Accept(0x4004d862a0, 0x665e1c, 0x0, 0xfe5e60, 0x40024ac5c8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/tcpsock.go:261 +0x5c crypto/tls.(*listener).Accept(0x4004d862b8, 0x4004d84b70, 0x11ae500, 0x4002554cc0, 0xf59ac0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/crypto/tls/tls.go:67 +0x34 net/http.(*Server).Serve(0x4002554c40, 0x12ec710, 0x4004d862b8, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/server.go:2981 +0x234 github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer.func1(0x4002554c40, 0x12ec710, 0x4004d862b8, 0x400014c000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:754 +0x38 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).initHttpsServer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:753 +0x3d0 goroutine 146 [IO wait]: internal/poll.runtime_pollWait(0xffff607473c8, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004dd4198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004dd4180, 0x40047d3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004dd4180, 0x40047d3000, 0x1000, 0x1000, 0x40047f4be0, 0x469b10, 0x4004d98000) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010bc0, 0x40047d3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x40047fea20, 0x40047d3000, 0x1000, 0x1000, 0x40047f4cf8, 0x67867c, 0x40047bf4a0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40047f1800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x40047f1800, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x40047fea20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 147 [select]: net/http.(*persistConn).writeLoop(0x40047fea20) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 54 [IO wait]: internal/poll.runtime_pollWait(0xffff61352b90, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40000f0418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40000f0400, 0x40024bb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40000f0400, 0x40024bb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x4000010670, 0x40024bb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x4001452120, 0x40024bb000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x40024b80c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x40024b80c0, 0xa, 0x0, 0x0, 0x0, 0x0, 0x4001420c38) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x40024b80c0, 0x0, 0x4001420c78, 0x655784, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x40045c0300) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x40045c0300, 0x40045c4400, 0x200, 0x200, 0x4, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x400457e4c0, 0x40045c4400, 0x200, 0x200, 0x400144a347, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x400457e4c0, 0x40045c4400, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x400457e500, 0x40045c4400, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x400457bb80, 0x4001460000, 0x4001420eb8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x400457bb80, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x400457bb80, 0xee0d40, 0x40014584b0, 0x12db068, 0x400457e500) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x40045be2d0, 0x4000083c80, 0x4004598960) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 57 [select]: net/http.(*persistConn).writeLoop(0x400474d0e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 95 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).buildTokenCleaner(0x40000e90a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:585 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:151 +0xa64 goroutine 96 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).delTokenCleaner(0x40000e90a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:474 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:152 +0xa84 goroutine 97 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).dropInstTokenCleaner(0x40000e90a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1175 +0xb8 created by github.com/couchbase/indexing/secondary/indexer.NewDDLServiceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:153 +0xaa4 goroutine 162 [sleep]: time.Sleep(0xdf8475800) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/time.go:193 +0xcc github.com/couchbase/indexing/secondary/indexer.(*schedTokenMonitor).updater(0x4004800d80) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:985 +0x3c created by github.com/couchbase/indexing/secondary/indexer.NewSchedTokenMonitor /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:936 +0x1dc goroutine 56 [IO wait]: internal/poll.runtime_pollWait(0xffff61352d60, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x40001b3f98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x40001b3f80, 0x40045c9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x40001b3f80, 0x40045c9000, 0x1000, 0x1000, 0x40047f9be0, 0x469b10, 0x4000000180) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x40014560f0, 0x40045c9000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400474d0e0, 0x40045c9000, 0x1000, 0x1000, 0x40047f9cf8, 0x67867c, 0x4004e626c0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4000083e00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).Peek(0x4000083e00, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:139 +0x40 net/http.(*persistConn).readLoop(0x400474d0e0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2083 +0x170 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 165 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40046578c0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 166 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).listenSnapshotReqs(0x40046578c0, 0x1) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:1296 +0x80 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:153 +0x26c goroutine 167 [chan receive]: github.com/couchbase/indexing/secondary/indexer.(*storageMgr).run(0x40046578c0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:172 +0x58 created by github.com/couchbase/indexing/secondary/indexer.NewStorageManager /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:157 +0x2a0 goroutine 168 [select]: github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).processCreateCommand(0x40000e90a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:1043 +0x138 created by github.com/couchbase/indexing/secondary/indexer.(*DDLServiceMgr).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/ddl_service_manager.go:177 +0x3c goroutine 169 [select]: github.com/couchbase/cbauth/metakv.doRunObserveChildren(0x4000112750, 0x1144559, 0x1b, 0x400445fe38, 0x4004bdd3e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:305 +0x248 github.com/couchbase/cbauth/metakv.(*store).runObserveChildren(...) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:263 github.com/couchbase/cbauth/metakv.RunObserveChildrenV2(0x1144559, 0x1b, 0x4004d321a0, 0x4004bdd3e0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:417 +0x88 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2.1(0x0, 0x0, 0x0, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1523 +0xe8 github.com/couchbase/indexing/secondary/common.(*RetryHelper).Run(0x400445ff80, 0x0, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/retry_helper.go:36 +0x98 github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens.func2(0x4004c2c720, 0x4004d321a0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1528 +0xa8 created by github.com/couchbase/indexing/secondary/manager/common.(*CommandListener).ListenTokens /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/manager/common/token.go:1516 +0x74 goroutine 174 [IO wait]: internal/poll.runtime_pollWait(0xffff607474b0, 0x72, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/runtime/netpoll.go:222 +0x44 internal/poll.(*pollDesc).wait(0x4004d44098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:87 +0x38 internal/poll.(*pollDesc).waitRead(...) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0x4004d44080, 0x4004d4e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/internal/poll/fd_unix.go:166 +0x180 net.(*netFD).Read(0x4004d44080, 0x4004d4e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/fd_posix.go:55 +0x44 net.(*conn).Read(0x400011a558, 0x4004d4e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/net.go:183 +0x74 net/http.(*persistConn).Read(0x400474d200, 0x4004d4e000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1922 +0x60 bufio.(*Reader).fill(0x4004c2d320) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:101 +0xf8 bufio.(*Reader).ReadSlice(0x4004c2d320, 0x100a, 0x0, 0x0, 0x400019e3b8, 0x456e60, 0x400019e438) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/bufio/bufio.go:360 +0x38 net/http/internal.readChunkLine(0x4004c2d320, 0x412b8c, 0x400019e478, 0x655784, 0x400019e4a8, 0x40b2f8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:122 +0x30 net/http/internal.(*chunkedReader).beginChunk(0x4004c75680) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:48 +0x2c net/http/internal.(*chunkedReader).Read(0x4004c75680, 0x4004beb600, 0x200, 0x200, 0x4, 0x0, 0x1) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/internal/chunked.go:93 +0x118 net/http.(*body).readLocked(0x4004800ec0, 0x4004beb600, 0x200, 0x200, 0x40047098c7, 0xe, 0x1c1c218) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:842 +0x54 net/http.(*body).Read(0x4004800ec0, 0x4004beb600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transfer.go:834 +0xec net/http.(*bodyEOFSignal).Read(0x4004800f00, 0x4004beb600, 0x200, 0x200, 0x0, 0x0, 0x0) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2764 +0xd8 encoding/json.(*Decoder).refill(0x4004d38580, 0x0, 0x400019e6b8) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:165 +0xd4 encoding/json.(*Decoder).readValue(0x4004d38580, 0x0, 0x0, 0x92d028) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:140 +0x1b8 encoding/json.(*Decoder).Decode(0x4004d38580, 0xee0d40, 0x4000129270, 0x12db068, 0x4004800f00) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/encoding/json/stream.go:63 +0x60 github.com/couchbase/cbauth/metakv.doRunObserveChildren.func1(0x4004c6ce10, 0x4004c2d4a0, 0x4004d50060) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:288 +0xfc created by github.com/couchbase/cbauth/metakv.doRunObserveChildren /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/cbauth/metakv/metakv.go:284 +0x1bc goroutine 172 [select]: net/http.(*persistConn).readLoop(0x400474d200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2203 +0x768 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1743 +0x99c goroutine 173 [select]: net/http.(*persistConn).writeLoop(0x400474d200) /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:2382 +0xa8 created by net/http.(*Transport).dialConn /home/couchbase/.cbdepscache/exploded/aarch64/go-1.16.5/go/src/net/http/transport.go:1744 +0x9bc goroutine 175 [runnable]: github.com/couchbase/indexing/secondary/common.(*ClusterInfoClient).watchClusterChanges(0x4004800dc0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1247 created by github.com/couchbase/indexing/secondary/common.NewClusterInfoClient /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/common/cluster_info.go:1226 +0x110 goroutine 176 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).run(0x4004657760) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:216 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:202 +0x578 goroutine 177 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).stopTokenCleaner(0x4004657760) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:610 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:203 +0x598 goroutine 178 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).orphanTokenMover(0x4004657760) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:660 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:204 +0x5b8 goroutine 179 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*schedIndexCreator).keyspaceMonitor(0x4004657760) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:750 created by github.com/couchbase/indexing/secondary/indexer.NewSchedIndexCreator /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/sched_index_creator.go:205 +0x5d8 goroutine 180 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).run(0x4004d3aa00) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:242 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:190 +0x4f4 goroutine 181 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*ServiceMgr).initService(0x4004d3aa00, 0x0) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:196 created by github.com/couchbase/indexing/secondary/indexer.NewRebalanceMgr /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/rebalance_service_manager.go:191 +0x51c goroutine 182 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).monitorKVNodes(0x400014c000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:9588 created by github.com/couchbase/indexing/secondary/indexer.NewIndexer /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:458 +0x1950 goroutine 183 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenWorkerMsgs(0x400014c000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1085 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:938 +0x44 goroutine 184 [runnable]: github.com/couchbase/indexing/secondary/indexer.(*indexer).listenAdminMsgs(0x400014c000) /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:1013 created by github.com/couchbase/indexing/secondary/indexer.(*indexer).run /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:939 +0x64 goroutine 185 [runnable]: github.com/couchbase/plasma.RunMemQuotaTuner() /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/plasma/mem.go:233 created by github.com/couchbase/indexing/secondary/indexer.newPlasmaSlice /home/couchbase/jenkins/workspace/couchbase-server-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/plasma_slice.go:209 +0xe60