Details
-
Bug
-
Resolution: Fixed
-
Test Blocker
-
5.5.0
Description
Ubuntu longevity - 5.5.0-1991 - rebalance got stuck for more than 8 hours:
2018-02-27T15:43:27.772-08:00, ns_cluster:3:info:message(ns_1@172.23.105.63) - Node ns_1@172.23.105.63 joined cluster
|
2018-02-27T15:43:37.398-08:00, ns_orchestrator:4:info:message(ns_1@172.23.106.14) - Starting rebalance, KeepNodes = ['ns_1@172.23.104.41','ns_1@172.23.105.60',
|
'ns_1@172.23.105.61','ns_1@172.23.105.62',
|
'ns_1@172.23.105.63','ns_1@172.23.105.83',
|
'ns_1@172.23.106.14','ns_1@172.23.106.213',
|
'ns_1@172.23.106.96','ns_1@172.23.99.253'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes
|
|
2018-02-27T15:43:38.079-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.63) - Deleting old data files of bucket "default"
|
2018-02-27T15:43:38.079-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.63) - Deleting old data files of bucket "WAREHOUSE"
|
2018-02-27T15:43:38.079-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.63) - Deleting old data files of bucket "STOCK"
|
2018-02-27T15:43:38.079-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.63) - Deleting old data files of bucket "ORDER_LINE"
|
2018-02-27T15:43:38.079-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.63) - Deleting old data files of bucket "ORDERS"
|
2018-02-27T15:43:38.079-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.63) - Deleting old data files of bucket "NEW_ORDER"
|
2018-02-27T15:43:38.079-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.63) - Deleting old data files of bucket "ITEM"
|
2018-02-27T15:43:38.079-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.63) - Deleting old data files of bucket "HISTORY"
|
2018-02-27T15:43:38.079-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.63) - Deleting old data files of bucket "DISTRICT"
|
2018-02-27T15:43:38.079-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.63) - Deleting old data files of bucket "CUSTOMER"
|
2018-02-27T15:43:38.080-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.83) - Deleting old data files of bucket "default"
|
2018-02-27T15:43:38.080-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.83) - Deleting old data files of bucket "WAREHOUSE"
|
2018-02-27T15:43:38.080-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.83) - Deleting old data files of bucket "STOCK"
|
2018-02-27T15:43:38.080-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.83) - Deleting old data files of bucket "ORDER_LINE"
|
2018-02-27T15:43:38.080-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.83) - Deleting old data files of bucket "ORDERS"
|
2018-02-27T15:43:38.080-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.83) - Deleting old data files of bucket "NEW_ORDER"
|
2018-02-27T15:43:38.080-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.83) - Deleting old data files of bucket "ITEM"
|
2018-02-27T15:43:38.081-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.83) - Deleting old data files of bucket "HISTORY"
|
2018-02-27T15:43:38.081-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.83) - Deleting old data files of bucket "DISTRICT"
|
2018-02-27T15:43:38.081-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.83) - Deleting old data files of bucket "CUSTOMER"
|
2018-02-27T15:43:38.976-08:00, ns_rebalancer:0:info:message(ns_1@172.23.106.14) - Started rebalancing bucket WAREHOUSE
|
2018-02-27T15:43:39.095-08:00, ns_memcached:0:info:message(ns_1@172.23.105.83) - Bucket "WAREHOUSE" loaded on node 'ns_1@172.23.105.83' in 0 seconds.
|
2018-02-27T15:43:39.124-08:00, ns_memcached:0:info:message(ns_1@172.23.105.63) - Bucket "WAREHOUSE" loaded on node 'ns_1@172.23.105.63' in 0 seconds.
|
2018-02-27T15:43:40.179-08:00, ns_vbucket_mover:0:info:message(ns_1@172.23.106.14) - Bucket "WAREHOUSE" rebalance does not seem to be swap rebalance
|
2018-02-27T15:43:48.540-08:00, ns_config:0:info:message(ns_1@172.23.106.14) - Conflicting configuration changes to field cbas_memory_quota:
|
2352 and
|
2412, choosing the former. (repeated 1 times)
|
2018-02-27T15:43:55.702-08:00, compaction_new_daemon:0:warning:message(ns_1@172.23.105.62) - Compactor for vbucket `default/343` (pid [{type,vbucket},
|
{name,<<"default/343">>},
|
{important,false},
|
{fa,
|
{#Fun<compaction_new_daemon.13.106246998>,
|
[<<"default">>,
|
{343,<<"default/343">>},
|
{config,
|
{30,undefined},
|
{30,undefined},
|
undefined,false,false,
|
{daemon_config,30,131072,
|
20971520}},
|
false,
|
{1519515817,0,false}]}}]) terminated unexpectedly (ignoring this): badarg
|
2018-02-27T15:44:18.862-08:00, compaction_new_daemon:0:warning:message(ns_1@172.23.106.14) - Compactor for vbucket `default/347` (pid [{type,vbucket},
|
{name,<<"default/347">>},
|
{important,false},
|
{fa,
|
{#Fun<compaction_new_daemon.13.106246998>,
|
[<<"default">>,
|
{347,<<"default/347">>},
|
{config,
|
{30,undefined},
|
{30,undefined},
|
undefined,false,false,
|
{daemon_config,30,131072,
|
20971520}},
|
false,
|
{1519515817,0,false}]}}]) terminated unexpectedly (ignoring this): badarg
|
2018-02-27T15:44:18.868-08:00, compaction_new_daemon:0:warning:message(ns_1@172.23.106.14) - Compactor for vbucket `default/351` (pid [{type,vbucket},
|
{name,<<"default/351">>},
|
{important,false},
|
{fa,
|
{#Fun<compaction_new_daemon.13.106246998>,
|
[<<"default">>,
|
{351,<<"default/351">>},
|
{config,
|
{30,undefined},
|
{30,undefined},
|
undefined,false,false,
|
{daemon_config,30,131072,
|
20971520}},
|
false,
|
{1519515817,0,false}]}}]) terminated unexpectedly (ignoring this): badarg
|
2018-02-27T15:44:18.879-08:00, compaction_new_daemon:0:warning:message(ns_1@172.23.106.14) - Compactor for vbucket `default/355` (pid [{type,vbucket},
|
{name,<<"default/355">>},
|
{important,false},
|
{fa,
|
{#Fun<compaction_new_daemon.13.106246998>,
|
[<<"default">>,
|
{355,<<"default/355">>},
|
{config,
|
{30,undefined},
|
{30,undefined},
|
undefined,false,false,
|
{daemon_config,30,131072,
|
20971520}},
|
false,
|
{1519515817,0,false}]}}]) terminated unexpectedly (ignoring this): badarg
|
2018-02-27T15:44:18.882-08:00,
|
...
|
compaction_new_daemon:0:warning:message(ns_1@172.23.106.14) - Compactor for vbucket `default/351` (pid [{type,vbucket},
|
{name,<<"default/351">>},
|
{important,false},
|
{fa,
|
{#Fun<compaction_new_daemon.13.106246998>,
|
[<<"default">>,
|
{351,<<"default/351">>},
|
{config,
|
{30,undefined},
|
{30,undefined},
|
undefined,false,false,
|
{daemon_config,30,131072,
|
20971520}},
|
false,
|
{1519515817,0,false}]}}]) terminated unexpectedly (ignoring this): badarg
|
2018-02-27T15:53:26.870-08:00, compaction_new_daemon:0:warning:message(ns_1@172.23.106.14) - Compactor for vbucket `default/355` (pid [{type,vbucket},
|
{name,<<"default/355">>},
|
{important,false},
|
{fa,
|
{#Fun<compaction_new_daemon.13.106246998>,
|
[<<"default">>,
|
{355,<<"default/355">>},
|
{config,
|
{30,undefined},
|
{30,undefined},
|
undefined,false,false,
|
{daemon_config,30,131072,
|
20971520}},
|
false,
|
{1519515817,0,false}]}}]) terminated unexpectedly (ignoring this): badarg
|
2018-02-27T15:53:26.873-08:00, compaction_new_daemon:0:warning:message(ns_1@172.23.106.14) - Compactor for vbucket `default/359` (pid [{type,vbucket},
|
{name,<<"default/359">>},
|
{important,false},
|
{fa,
|
{#Fun<compaction_new_daemon.13.106246998>,
|
[<<"default">>,
|
{359,<<"default/359">>},
|
{config,
|
{30,undefined},
|
{30,undefined},
|
undefined,false,false,
|
{daemon_config,30,131072,
|
20971520}},
|
false,
|
{1519515817,0,false}]}}]) terminated unexpectedly (ignoring this): badarg
|
2018-02-27T15:53:46.680-08:00, compaction_new_daemon:0:warning:message(ns_1@172.23.105.62) - Compactor for vbucket `default/343` (pid [{type,vbucket},
|
{name,<<"default/343">>},
|
{important,false},
|
{fa,
|
{#Fun<compaction_new_daemon.13.106246998>,
|
[<<"default">>,
|
{343,<<"default/343">>},
|
{config,
|
{30,undefined},
|
{30,undefined},
|
undefined,false,false,
|
{daemon_config,30,131072,
|
20971520}},
|
false,
|
{1519515817,0,false}]}}]) terminated unexpectedly (ignoring this): badarg
|
2018-02-27T15:53:52.518-08:00, ns_rebalancer:0:info:message(ns_1@172.23.106.14) - Started rebalancing bucket default
|
2018-02-27T15:53:52.637-08:00, ns_memcached:0:info:message(ns_1@172.23.105.83) - Bucket "default" loaded on node 'ns_1@172.23.105.83' in 0 seconds.
|
2018-02-27T15:53:52.642-08:00, ns_memcached:0:info:message(ns_1@172.23.105.63) - Bucket "default" loaded on node 'ns_1@172.23.105.63' in 0 seconds.
|
2018-02-27T15:53:53.751-08:00, ns_vbucket_mover:0:info:message(ns_1@172.23.106.14) - Bucket "default" rebalance does not seem to be swap rebalance
|
2018-02-27T23:19:44.322-08:00, ns_orchestrator:0:critical:message(ns_1@172.23.106.14) - Rebalance exited with reason {badmatch,
|
{leader_activities_error,
|
{default,rebalance},
|
quorum_lost}}
|
Attachments
Issue Links
- is triggered by
-
MB-28470 CheckpointManager::getNumItemsForCursor is imprecise
- Closed
For Gerrit Dashboard: MB-28431 | ||||||
---|---|---|---|---|---|---|
# | Subject | Branch | Project | Status | CR | V |
90347,3 | MB-28431: Avoid getNumItemsForCursor() when getting items to persist | master | kv_engine | Status: MERGED | +2 | +1 |