Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Critical
-
Resolution: Fixed
-
Affects Version/s: 6.6.0
-
Fix Version/s: 6.6.0
-
Component/s: analytics
-
Labels:
-
Environment:Enterprise Edition 6.6.0 build 7873 ‧ IPv4
-
Triage:Untriaged
-
Operating System:Centos 64-bit
-
Link to Log File, atop/blg, CBCollectInfo, Core dump:
-
Story Points:1
-
Is this a Regression?:Yes
-
Sprint:CX Sprint 207
Description
Rebalance of analytics node fails with :
Add and remove analytics node and then rebalance:
[2020-07-12T14:37:30-07:00, sequoiatools/couchbase-cli:6.5:b52b88] server-add -c 172.23.108.103:8091 --server-add https://172.23.99.20 -u Administrator -p password --server-add-username Administrator --server-add-password password --services analytics
|
[2020-07-12T14:38:12-07:00, sequoiatools/couchbase-cli:6.5:afaf7f] rebalance -c 172.23.108.103:8091 --server-remove 172.23.97.239 -u Administrator -p password
|
→
|
|
|
Error occurred on container - sequoiatools/couchbase-cli:6.5:[rebalance -c 172.23.108.103:8091 --server-remove 172.23.97.239 -u Administrator -p password]
|
|
|
docker logs afaf7f
|
docker start afaf7f
|
|
|
sWARNING: couchbase-cli version 6.5.0-3216-enterprise does not match couchbase server version 6.6.0-7873-enterprise
|
*Unable to display progress bar on this os
|
JERROR: Rebalance failed. See logs for detailed reason. You can try again.
|
[{'ns_1@172.23.97.122',
|
{exit,
|
{{nodedown,'ns_1@172.23.97.122'},
|
{gen_server,call,
|
[{ns_config_rep,'ns_1@172.23.97.122'},
|
synchronize_everything,infinity]}}}}]
|
[ns_server:error,2020-07-10T10:33:06.660-07:00,ns_1@172.23.108.103:service_rebalancer-cbas<0.9798.462>:service_rebalancer:run_rebalance_worker:125]Worker terminated abnormally: {'EXIT',<0.9793.462>,
|
{rebalance_failed,
|
{service_error,
|
<<"Rebalance d8e520f7fd4f00e1c531f8df5540817e failed: HYR0080: Cannot create index because it already exists">>}}}
|
[user:error,2020-07-10T10:33:06.663-07:00,ns_1@172.23.108.103:<0.24054.249>:ns_orchestrator:log_rebalance_completion:1445]Rebalance exited with reason {service_rebalance_failed,cbas,
|
{worker_died,
|
{'EXIT',<0.9793.462>,
|
{rebalance_failed,
|
{service_error,
|
<<"Rebalance d8e520f7fd4f00e1c531f8df5540817e failed: HYR0080: Cannot create index because it already exists">>}}}}}.
|
Rebalance Operation Id = b23e6cee01bee5b04faf7cc2bcf5053c
|
[ns_server:error,2020-07-10T11:01:36.380-07:00,ns_1@172.23.108.103:service_rebalancer-cbas<0.11839.480>:service_rebalancer:run_rebalance_worker:125]Worker terminated abnormally: {'EXIT',<0.696.480>,
|
{rebalance_failed,
|
{service_error,
|
<<"Rebalance 4e723c73ef045aa36646691a7e6045d9 failed: HYR0080: Cannot create index because it already exists">>}}}
|
|
Attachments
Issue Links
- relates to
-
MB-40472 [CX] Ensure that file references are always resolved on NC
-
- Open
-
- links to
Activity
| Assignee | Till Westmann [ till ] | Murtadha Hubail [ murtadha.hubail ] |
| Labels | system-test | system-test triaged |
| Rank | Ranked higher |
| Sprint | CX Sprint 207 [ 1145 ] |
| Rank | Ranked lower |
| Status | Open [ 1 ] | In Progress [ 3 ] |
| Remote Link | This issue links to "*DB review (Web Link)" [ 20236 ] |
| Link | This issue blocks MB-38724 [ MB-38724 ] |
| Labels | system-test triaged | approved-for-6.6.0 system-test triaged |
| Assignee | Murtadha Hubail [ murtadha.hubail ] | Ritam Sharma [ ritam.sharma ] |
| Resolution | Fixed [ 1 ] | |
| Status | In Progress [ 3 ] | Resolved [ 5 ] |
| VERIFICATION STEPS | Verified and closed for 6.6.0-7883. System testing run did not show any exception for 3 days of test run. | |
| Status | Resolved [ 5 ] | Closed [ 6 ] |
This issue happened because some rebalance was interrupted (e.g. cancelled or a crash happened in the middle of the rebalance) and that led to some unnecessary data existing from the cancelled rebalance. When the next rebalance came, we have a cleanup step that should've deleted the left over data. However, the cleanup process was not pointing to the correct file that should've been deleted because the file path wasn't resolved on the local node. This should be fixed now.