Critical Description
I was playing around with rebalance (rebalance out of the F of a D,D+F,F cluster), and mid-way through, I hit the Stop Rebalance button.
But, even though the ns-server UI correctly showed indications that rebalance activity had stopped, I noticed that the cbft service was still incorrectly performing pindex rebalance orchestration work, seeing cbft emit occasional progress log sections like...
2016-03-25T08:57:29.639-07:00 [INFO] index: default_index_1, #175 [TotMoverAssignPartitionOk: 28 -> 29]
|
2016-03-25T08:57:29.639-07:00 [INFO] index: default_index_1, #176 [TotMoverLoop: 30 -> 31]
|
2016-03-25T08:57:29.640-07:00 [INFO] %%% c9d504c016db2ef6d631d3a6e1499f3b ebe95c7325dc601fbbe37d9fc246a5cb
|
% default_index_1_21d243150d56243e_0a44bddb . .
|
% default_index_1_21d243150d56243e_0ffd4517 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_24e7ea2d . .
|
% default_index_1_21d243150d56243e_27184a97 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_321cbb28 . .
|
% default_index_1_21d243150d56243e_348f5c3c 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_37fdcf3c . .
|
% default_index_1_21d243150d56243e_3cf74ad7 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_4728ab89 . .
|
% default_index_1_21d243150d56243e_4a7ce4ca 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_4cdba0cf . .
|
% default_index_1_21d243150d56243e_54127e67 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_577e9974 . .
|
% default_index_1_21d243150d56243e_596cc6a1 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_5b1351c0 . .
|
% default_index_1_21d243150d56243e_5c5f941e 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_63ef8109 . .
|
% default_index_1_21d243150d56243e_76ac2a42 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_77082885 . .
|
% default_index_1_21d243150d56243e_8b80958a 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_9284f0ad . .
|
% default_index_1_21d243150d56243e_9e13fa43 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_a4a23588 . .
|
% default_index_1_21d243150d56243e_b7ff6b68 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_cd5a61e8 . .
|
% default_index_1_21d243150d56243e_d6109bff 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_d8c75a95 . .
|
% default_index_1_21d243150d56243e_d90ca3e7 0 . 1 .
|
% default_index_1_21d243150d56243e_dee61dfa . .
|
% default_index_1_21d243150d56243e_e64340fa 0 + 100.0% 1 .
|
% default_index_1_21d243150d56243e_ee590b16 . .
|
% default_index_1_21d243150d56243e_f17535e5 0 + 100.0% 1 -
|
2016-03-25T08:57:29.641-07:00 [INFO] index: default_index_1, #177 [TotRunSupplyMovesFeedingDone: 28 -> 29]
|
2016-03-25T08:57:29.641-07:00 [INFO] index: default_index_1, #178 [TotRunSupplyMovesLoop: 29 -> 30]
|
2016-03-25T08:57:29.642-07:00 [INFO] index: default_index_1, #179 [TotRunSupplyMovesFeeding: 29 -> 30]
|
2016-03-25T08:57:29.643-07:00 [INFO] index: default_index_1, #180 [TotMoverAssignPartition: 29 -> 30]
|
2016-03-25T08:57:29.643-07:00 [INFO] assignPIndex: index: default_index_1, pindex: default_index_1_21d243150d56243e_d90ca3e7, node: c9d504c016db2ef6d631d3a6e1499f3b, state: "primary", op: add
|
2016-03-25T08:57:29.644-07:00 [INFO] index: default_index_1, #181 [TotMoverAssignPartition: 30 -> 31]
|
2016-03-25T08:57:29.644-07:00 [INFO] assignPIndex: index: default_index_1, pindex: default_index_1_21d243150d56243e_e64340fa, node: ebe95c7325dc601fbbe37d9fc246a5cb, state: "", op: del
|
2016-03-25T08:57:29.651-07:00 [INFO] cfg_metakv: Set, key: planPIndexes, cas: 1, split: false, nodeUUID: ebe95c7325dc601fbbe37d9fc246a5cb
|
2016-03-25T08:57:29.651-07:00 [INFO] cfg_metakv: Set path: /fts/cbgt/cfg/planPIndexes
|
2016-03-25T08:57:29.659-07:00 [INFO] %%% c9d504c016db2ef6d631d3a6e1499f3b ebe95c7325dc601fbbe37d9fc246a5cb
|
% default_index_1_21d243150d56243e_0a44bddb . .
|
% default_index_1_21d243150d56243e_0ffd4517 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_24e7ea2d . .
|
% default_index_1_21d243150d56243e_27184a97 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_321cbb28 . .
|
% default_index_1_21d243150d56243e_348f5c3c 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_37fdcf3c . .
|
% default_index_1_21d243150d56243e_3cf74ad7 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_4728ab89 . .
|
% default_index_1_21d243150d56243e_4a7ce4ca 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_4cdba0cf . .
|
% default_index_1_21d243150d56243e_54127e67 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_577e9974 . .
|
% default_index_1_21d243150d56243e_596cc6a1 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_5b1351c0 . .
|
% default_index_1_21d243150d56243e_5c5f941e 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_63ef8109 . .
|
% default_index_1_21d243150d56243e_76ac2a42 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_77082885 . .
|
% default_index_1_21d243150d56243e_8b80958a 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_9284f0ad . .
|
% default_index_1_21d243150d56243e_9e13fa43 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_a4a23588 . .
|
% default_index_1_21d243150d56243e_b7ff6b68 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_cd5a61e8 . .
|
% default_index_1_21d243150d56243e_d6109bff 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_d8c75a95 . .
|
% default_index_1_21d243150d56243e_d90ca3e7 0 + 1 .
|
% default_index_1_21d243150d56243e_dee61dfa . .
|
% default_index_1_21d243150d56243e_e64340fa 0 + 100.0% 1 .
|
% default_index_1_21d243150d56243e_ee590b16 . .
|
% default_index_1_21d243150d56243e_f17535e5 0 + 100.0% 1 -
|
2016-03-25T08:57:29.662-07:00 [INFO] cfg_metakv: metaKVCallback, path: /fts/cbgt/cfg/planPIndexes, key: planPIndexes, deletion: false
|
2016-03-25T08:57:29.665-07:00 [INFO] janitor: awakes, op: kick, msg: cfg changed, key: planPIndexes
|
2016-03-25T08:57:29.669-07:00 [INFO] cfg_metakv: Set, key: planPIndexes, cas: 1, split: false, nodeUUID: ebe95c7325dc601fbbe37d9fc246a5cb
|
2016-03-25T08:57:29.674-07:00 [INFO] cfg_metakv: Set path: /fts/cbgt/cfg/planPIndexes
|
2016-03-25T08:57:29.677-07:00 [INFO] janitor: pindexes to remove: 0
|
2016-03-25T08:57:29.677-07:00 [INFO] janitor: pindexes to add: 0
|
2016-03-25T08:57:29.677-07:00 [INFO] janitor: feeds to remove: 0
|
2016-03-25T08:57:29.677-07:00 [INFO] janitor: feeds to add: 0
|
2016-03-25T08:57:29.680-07:00 [INFO] index: default_index_1, #182 [TotMoverAssignPartitionOk: 29 -> 30]
|
2016-03-25T08:57:29.682-07:00 [INFO] %%% c9d504c016db2ef6d631d3a6e1499f3b ebe95c7325dc601fbbe37d9fc246a5cb
|
% default_index_1_21d243150d56243e_0a44bddb . .
|
% default_index_1_21d243150d56243e_0ffd4517 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_24e7ea2d . .
|
% default_index_1_21d243150d56243e_27184a97 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_321cbb28 . .
|
% default_index_1_21d243150d56243e_348f5c3c 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_37fdcf3c . .
|
% default_index_1_21d243150d56243e_3cf74ad7 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_4728ab89 . .
|
% default_index_1_21d243150d56243e_4a7ce4ca 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_4cdba0cf . .
|
% default_index_1_21d243150d56243e_54127e67 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_577e9974 . .
|
% default_index_1_21d243150d56243e_596cc6a1 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_5b1351c0 . .
|
% default_index_1_21d243150d56243e_5c5f941e 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_63ef8109 . .
|
% default_index_1_21d243150d56243e_76ac2a42 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_77082885 . .
|
% default_index_1_21d243150d56243e_8b80958a 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_9284f0ad . .
|
% default_index_1_21d243150d56243e_9e13fa43 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_a4a23588 . .
|
% default_index_1_21d243150d56243e_b7ff6b68 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_cd5a61e8 . .
|
% default_index_1_21d243150d56243e_d6109bff 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_d8c75a95 . .
|
% default_index_1_21d243150d56243e_d90ca3e7 0 + 1 .
|
% default_index_1_21d243150d56243e_dee61dfa . .
|
% default_index_1_21d243150d56243e_e64340fa 0 + 100.0% 1 -
|
% default_index_1_21d243150d56243e_ee590b16 . .
|
% default_index_1_21d243150d56243e_f17535e5 0 + 100.0% 1 -
|
2016-03-25T08:57:29.682-07:00 [INFO] index: default_index_1, #183 [TotMoverLoop: 31 -> 32]
|
2016-03-25T08:57:29.683-07:00 [INFO] index: default_index_1, #184 [TotRunSupplyMovesFeedingDone: 29 -> 30]
|
2016-03-25T08:57:29.684-07:00 [INFO] cfg_metakv: metaKVCallback, path: /fts/cbgt/cfg/planPIndexes, key: planPIndexes, deletion: false
|
2016-03-25T08:57:29.684-07:00 [INFO] janitor: awakes, op: kick, msg: cfg changed, key: planPIndexes
|
2016-03-25T08:57:29.685-07:00 [INFO] index: default_index_1, #185 [TotRunSupplyMovesLoop: 30 -> 31]
|
2016-03-25T08:57:29.689-07:00 [INFO] index: default_index_1, #186 [TotRunSupplyMovesFeeding: 30 -> 31]
|
2016-03-25T08:57:29.697-07:00 [INFO] janitor: pindexes to remove: 1
|
2016-03-25T08:57:29.697-07:00 [INFO] &{Name:default_index_1_21d243150d56243e_e64340fa UUID:133d4cca186ba82d IndexType:fulltext-index IndexName:default_index_1 IndexUUID:21d243150d56243e IndexParams:{} SourceType:couchbase SourceName:default SourceUUID: SourceParams:{"authPassword":"","authSaslPassword":"","authSaslUser":"","authUser":"default","clusterManagerBackoffFactor":0,"clusterManagerSleepInitMS":0,"clusterManagerSleepMaxMS":20000,"dataManagerBackoffFactor":0,"dataManagerSleepInitMS":0,"dataManagerSleepMaxMS":20000,"feedBufferAckThreshold":0,"feedBufferSizeBytes":0} SourcePartitions:416,417,418,419,420,421,422,423,424,425,426,427,428,429,430,431,432,433,434,435,436,437,438,439,440,441,442,443,444,445,446,447 Path:/Users/steveyen/work/membase/dev/couchbase-server.watson/ns_server/data/n_2/data/@fts/default_index_1_21d243150d56243e_e64340fa.pindex Impl:0xc8203cad20 Dest:0xc8202a2bd0 sourcePartitionsMap:map[416:true 418:true 421:true 424:true 442:true 446:true 417:true 428:true 435:true 438:true 444:true 423:true 447:true 425:true 429:true 437:true 439:true 441:true 426:true 431:true 436:true 419:true 422:true 433:true 440:true 443:true 430:true 434:true 445:true 420:true 427:true 432:true] m:{state:0 sema:0} closed:false}
|
2016-03-25T08:57:29.697-07:00 [INFO] janitor: pindexes to add: 0
|
2016-03-25T08:57:29.697-07:00 [INFO] janitor: removing pindex: default_index_1_21d243150d56243e_e64340fa
|
2016-03-25T08:57:29.698-07:00 [INFO] feed_dcp: close, name: default_index_1_21d243150d56243e
|
2016-03-25T08:57:29.698-07:00 [INFO] feed_dcp: OnError, name: default_index_1_21d243150d56243e: bucketName: default, bucketUUID: , err: pkt.Receive, err: read tcp 10.72.57.110:53412->10.72.57.110:12000: use of closed network connection
|
2016-03-25T08:57:29.698-07:00 [INFO] main: meh.OnFeedError, srcType: couchbase, err: pkt.Receive, err: read tcp 10.72.57.110:53412->10.72.57.110:12000: use of closed network connection
|
2016-03-25T08:57:29.703-07:00 [INFO] moss_herder: OnCloseStart, collection: 0xc8202784b0
|
2016-03-25T08:57:29.706-07:00 [INFO] moss_herder: OnClose, collection: 0xc8202784b0
|
2016-03-25T08:57:29.708-07:00 [INFO] janitor: feeds to remove: 0
|
2016-03-25T08:57:29.708-07:00 [INFO] janitor: feeds to add: 1
|
2016-03-25T08:57:29.708-07:00 [INFO] default_index_1_21d243150d56243e
|
2016-03-25T08:57:29.708-07:00 [INFO] feed_dcp: NewDCPFeed, name: default_index_1_21d243150d56243e, indexName: default_index_1
|
2016-03-25T08:57:29.709-07:00 [INFO] feed_dcp: start, name: default_index_1_21d243150d56243e
|