Details
-
Bug
-
Resolution: Fixed
-
Critical
-
Cheshire-Cat
-
Untriaged
-
-
1
-
Yes
Description
Build : 7.0.0-3472
Test : -test tests/integration/test_allFeatures_madhatter_durability.yml -scope tests/integration/scope_Xattrs_Madhatter.yml
Scale : 3
Iteration : 1st
This issue is seen in the MH longevity system test (without collections)
At 2020-10-21T15:40:21, a rebalance operation was started to add a new eventing node 172.23.96.48 to the cluster. At 2020-10-21T15:56:42, this rebalance operation failed with the following error -
[ns_server:error,2020-10-21T15:56:35.534-07:00,ns_1@172.23.97.74:service_rebalancer-index<0.24547.32>:service_rebalancer:run_rebalance_worker:125]Worker terminated abnormally: {'EXIT',<0.24750.32>,
|
{rebalance_failed,
|
{service_error,
|
<<"Index default.sg_channels_x1 already exists">>}}}
|
[user:error,2020-10-21T15:56:35.536-07:00,ns_1@172.23.97.74:<0.23070.0>:ns_orchestrator:log_rebalance_completion:1398]Rebalance exited with reason {service_rebalance_failed,index,
|
{worker_died,
|
{'EXIT',<0.24750.32>,
|
{rebalance_failed,
|
{service_error,
|
<<"Index default.sg_channels_x1 already exists">>}}}}}.
|
Rebalance Operation Id = d32f3393b0ca65d4014dbda9eff16c0a
|
The following is seen in the indexer logs of 172.23.123.32 around the same time :
2020-10-21T15:56:30.866-07:00 [Info] Rebalancer::decodeTransferToken TransferToken TransferTokend8:9b:36:a3:ba:bb:fa:8e MasterId: 84ae849a00176d270c134a7abdd5d2bc SourceId: 7a2d9b5bcad6f9086a648970df7a1889 DestId: 2c49272f0070b26c10f6ff85c86921e0 RebalId: 432e7328f0bd839741e9b706f2381222 State: TransferTokenDeleted BuildSource: Dcp TransferMode: Move InstId: 8199086800912608841 RealInstId: 9009857079511335752 Partitions: [3] Versions: [1] Inst:
|
InstId: 9009857079511335752
|
Defn: DefnId: 3956965771300335729 Name: o2_result_rating Using: plasma Bucket: DISTRICT Scope/Id: _default/0 Collection/Id: _default/0 IsPrimary: false NumReplica: 0 InstVersion: 1
|
SecExprs: <ud>([`rating` `result`])</ud>
|
Desc: [false false]
|
PartitionScheme: KEY
|
HashScheme: CRC32 PartitionKeys: [`rating`] WhereExpr: <ud>()</ud> RetainDeletedXATTR: false
|
State: INDEX_STATE_ACTIVE
|
RState: RebalActive
|
Stream: NIL_STREAM
|
Version: 0
|
ReplicaId: 0
|
PartitionContainer: <nil>
|
2020-10-21T15:56:30.888-07:00 [Info] Rebalancer::publishTransferTokenBatch Registered Transfer Token In Metakv [TransferToken75:f:cd:de:30:f1:84:c7 TransferTokena:49:75:86:f6:b7:17:b3 TransferToken55:ee:d4:c7:d8:23:24:93]
|
2020-10-21T15:56:30.985-07:00 [Info] Rebalancer::processTokens Received empty or deleted transfer token /indexing/rebalance/TransferTokend8:9b:36:a3:ba:bb:fa:8e
|
2020-10-21T15:56:30.985-07:00 [Info] Rebalancer::decodeTransferToken TransferToken TransferToken75:f:cd:de:30:f1:84:c7 MasterId: 84ae849a00176d270c134a7abdd5d2bc SourceId: a157b09656e3f4ee220209cca524938d DestId: 84ae849a00176d270c134a7abdd5d2bc RebalId: 432e7328f0bd839741e9b706f2381222 State: TransferTokenCreated BuildSource: Dcp TransferMode: Move InstId: 5976024996375597664 RealInstId: 0 Partitions: [0] Versions: [1] Inst:
|
InstId: 5976024996375597664
|
Defn: DefnId: 9049312182255072788 Name: sg_channels_x1 Using: plasma Bucket: default Scope/Id: _default/0 Collection/Id: _default/0 IsPrimary: false NumReplica: 1 InstVersion: 1
|
SecExprs: <ud>([(all (array [(`op`.`name`), least((((meta().`xattrs`).`_sync`).`sequence`), ((`op`.`val`).`seq`)), ifmissing(((`op`.`val`).`rev`), null), ifmissing(((`op`.`val`).`del`), null)] for `op` in object_pairs((((meta().`xattrs`).`_sync`).`channels`)) end)) (((meta().`xattrs`).`_sync`).`rev`) (((meta().`xattrs`).`_sync`).`sequence`) (((meta().`xattrs`).`_sync`).`flags`)])</ud>
|
Desc: [false false false false]
|
PartitionScheme: SINGLE
|
HashScheme: CRC32 PartitionKeys: [] WhereExpr: <ud>()</ud> RetainDeletedXATTR: true
|
State: INDEX_STATE_ACTIVE
|
RState: RebalActive
|
Stream: NIL_STREAM
|
Version: 0
|
ReplicaId: 1
|
PartitionContainer: <nil>
|
2020-10-21T15:56:31.014-07:00 [Info] lifecycleMgr.dispatchRequest: op OPCODE_CREATE_INDEX_REBAL elapsed 160.271µs len(expediates) 0 len(incomings) 0 len(outgoings) 0
|
2020-10-21T15:56:31.014-07:00 [Error] Rebalancer::processTokenAsDest Error cloning index 127.0.0.1:9102/createIndexRebalance Index default.sg_channels_x1 already exists
|
2020-10-21T15:56:31.067-07:00 [Info] Rebalancer::decodeTransferToken TransferToken TransferTokena:49:75:86:f6:b7:17:b3 MasterId: 84ae849a00176d270c134a7abdd5d2bc SourceId: b33ce32a9760d1ddf3381f1c8e8f0689 DestId: 2c49272f0070b26c10f6ff85c86921e0 RebalId: 432e7328f0bd839741e9b706f2381222 State: TransferTokenCreated BuildSource: Dcp TransferMode: Move InstId: 13229364255755010755 RealInstId: 0 Partitions: [0] Versions: [1] Inst:
|
InstId: 13229364255755010755
|
Defn: DefnId: 2811334251040763543 Name: #primary Using: plasma Bucket: default Scope/Id: _default/0 Collection/Id: _default/0 IsPrimary: true NumReplica: 2 InstVersion: 1
|
SecExprs: <ud>([])</ud>
|
Desc: []
|
PartitionScheme: SINGLE
|
HashScheme: CRC32 PartitionKeys: [] WhereExpr: <ud>()</ud> RetainDeletedXATTR: false
|
State: INDEX_STATE_ACTIVE
|
RState: RebalActive
|
Stream: NIL_STREAM
|
Version: 0
|
ReplicaId: 2
|
PartitionContainer: <nil>
|
Another point here is that during rebalance of a non-indexer node, GSI indexes shouldn't be moved around.
This issue was seen 4 times so far in 5-6 hrs of the test run. Also, this is a regression, as this issue wasn't seen until now. The last time the test was run was with 7.0.0-3342