Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Fixed
Priority: Test Blocker
Fix Version/s: 5.5.0
Affects Version/s: 5.5.0
Component/s: couchbase-bucket
Labels:
- DP
- system-test

Triage:
Untriaged
Link to Log File, atop/blg, CBCollectInfo, Core dump:
https://s3.amazonaws.com/bugdb/jira/feb28/collectinfo-2018-02-28T185507-ns_1%40172.23.106.14.zip
Is this a Regression?:
Yes

Description

Ubuntu longevity - 5.5.0-1991 - rebalance got stuck for more than 8 hours:

2018-02-27T15:43:27.772-08:00, ns_cluster:3:info:message(ns_1@172.23.105.63) - Node ns_1@172.23.105.63 joined cluster

2018-02-27T15:43:37.398-08:00, ns_orchestrator:4:info:message(ns_1@172.23.106.14) - Starting rebalance, KeepNodes = ['ns_1@172.23.104.41','ns_1@172.23.105.60',

                                 'ns_1@172.23.105.61','ns_1@172.23.105.62',

                                 'ns_1@172.23.105.63','ns_1@172.23.105.83',

                                 'ns_1@172.23.106.14','ns_1@172.23.106.213',

                                 'ns_1@172.23.106.96','ns_1@172.23.99.253'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes

2018-02-27T15:43:38.079-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.63) - Deleting old data files of bucket "default"

2018-02-27T15:43:38.079-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.63) - Deleting old data files of bucket "WAREHOUSE"

2018-02-27T15:43:38.079-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.63) - Deleting old data files of bucket "STOCK"

2018-02-27T15:43:38.079-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.63) - Deleting old data files of bucket "ORDER_LINE"

2018-02-27T15:43:38.079-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.63) - Deleting old data files of bucket "ORDERS"

2018-02-27T15:43:38.079-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.63) - Deleting old data files of bucket "NEW_ORDER"

2018-02-27T15:43:38.079-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.63) - Deleting old data files of bucket "ITEM"

2018-02-27T15:43:38.079-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.63) - Deleting old data files of bucket "HISTORY"

2018-02-27T15:43:38.079-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.63) - Deleting old data files of bucket "DISTRICT"

2018-02-27T15:43:38.079-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.63) - Deleting old data files of bucket "CUSTOMER"

2018-02-27T15:43:38.080-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.83) - Deleting old data files of bucket "default"

2018-02-27T15:43:38.080-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.83) - Deleting old data files of bucket "WAREHOUSE"

2018-02-27T15:43:38.080-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.83) - Deleting old data files of bucket "STOCK"

2018-02-27T15:43:38.080-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.83) - Deleting old data files of bucket "ORDER_LINE"

2018-02-27T15:43:38.080-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.83) - Deleting old data files of bucket "ORDERS"

2018-02-27T15:43:38.080-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.83) - Deleting old data files of bucket "NEW_ORDER"

2018-02-27T15:43:38.080-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.83) - Deleting old data files of bucket "ITEM"

2018-02-27T15:43:38.081-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.83) - Deleting old data files of bucket "HISTORY"

2018-02-27T15:43:38.081-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.83) - Deleting old data files of bucket "DISTRICT"

2018-02-27T15:43:38.081-08:00, ns_storage_conf:0:info:message(ns_1@172.23.105.83) - Deleting old data files of bucket "CUSTOMER"

2018-02-27T15:43:38.976-08:00, ns_rebalancer:0:info:message(ns_1@172.23.106.14) - Started rebalancing bucket WAREHOUSE

2018-02-27T15:43:39.095-08:00, ns_memcached:0:info:message(ns_1@172.23.105.83) - Bucket "WAREHOUSE" loaded on node 'ns_1@172.23.105.83' in 0 seconds.

2018-02-27T15:43:39.124-08:00, ns_memcached:0:info:message(ns_1@172.23.105.63) - Bucket "WAREHOUSE" loaded on node 'ns_1@172.23.105.63' in 0 seconds.

2018-02-27T15:43:40.179-08:00, ns_vbucket_mover:0:info:message(ns_1@172.23.106.14) - Bucket "WAREHOUSE" rebalance does not seem to be swap rebalance

2018-02-27T15:43:48.540-08:00, ns_config:0:info:message(ns_1@172.23.106.14) - Conflicting configuration changes to field cbas_memory_quota:

2352 and

2412, choosing the former. (repeated 1 times)

2018-02-27T15:43:55.702-08:00, compaction_new_daemon:0:warning:message(ns_1@172.23.105.62) - Compactor for vbucket `default/343` (pid [{type,vbucket},

                                          {name,<<"default/343">>},

                                          {important,false},

                                          {fa,

                                           {#Fun<compaction_new_daemon.13.106246998>,

                                            [<<"default">>,

                                             {343,<<"default/343">>},

                                             {config,

                                              {30,undefined},

                                              {30,undefined},

                                              undefined,false,false,

                                              {daemon_config,30,131072,

                                               20971520}},

                                             false,

                                             {1519515817,0,false}]}}]) terminated unexpectedly (ignoring this): badarg

2018-02-27T15:44:18.862-08:00, compaction_new_daemon:0:warning:message(ns_1@172.23.106.14) - Compactor for vbucket `default/347` (pid [{type,vbucket},

                                          {name,<<"default/347">>},

                                          {important,false},

                                          {fa,

                                           {#Fun<compaction_new_daemon.13.106246998>,

                                            [<<"default">>,

                                             {347,<<"default/347">>},

                                             {config,

                                              {30,undefined},

                                              {30,undefined},

                                              undefined,false,false,

                                              {daemon_config,30,131072,

                                               20971520}},

                                             false,

                                             {1519515817,0,false}]}}]) terminated unexpectedly (ignoring this): badarg

2018-02-27T15:44:18.868-08:00, compaction_new_daemon:0:warning:message(ns_1@172.23.106.14) - Compactor for vbucket `default/351` (pid [{type,vbucket},

                                          {name,<<"default/351">>},

                                          {important,false},

                                          {fa,

                                           {#Fun<compaction_new_daemon.13.106246998>,

                                            [<<"default">>,

                                             {351,<<"default/351">>},

                                             {config,

                                              {30,undefined},

                                              {30,undefined},

                                              undefined,false,false,

                                              {daemon_config,30,131072,

                                               20971520}},

                                             false,

                                             {1519515817,0,false}]}}]) terminated unexpectedly (ignoring this): badarg

2018-02-27T15:44:18.879-08:00, compaction_new_daemon:0:warning:message(ns_1@172.23.106.14) - Compactor for vbucket `default/355` (pid [{type,vbucket},

                                          {name,<<"default/355">>},

                                          {important,false},

                                          {fa,

                                           {#Fun<compaction_new_daemon.13.106246998>,

                                            [<<"default">>,

                                             {355,<<"default/355">>},

                                             {config,

                                              {30,undefined},

                                              {30,undefined},

                                              undefined,false,false,

                                              {daemon_config,30,131072,

                                               20971520}},

                                             false,

                                             {1519515817,0,false}]}}]) terminated unexpectedly (ignoring this): badarg

2018-02-27T15:44:18.882-08:00,

...

compaction_new_daemon:0:warning:message(ns_1@172.23.106.14) - Compactor for vbucket `default/351` (pid [{type,vbucket},

                                          {name,<<"default/351">>},

                                          {important,false},

                                          {fa,

                                           {#Fun<compaction_new_daemon.13.106246998>,

                                            [<<"default">>,

                                             {351,<<"default/351">>},

                                             {config,

                                              {30,undefined},

                                              {30,undefined},

                                              undefined,false,false,

                                              {daemon_config,30,131072,

                                               20971520}},

                                             false,

                                             {1519515817,0,false}]}}]) terminated unexpectedly (ignoring this): badarg

2018-02-27T15:53:26.870-08:00, compaction_new_daemon:0:warning:message(ns_1@172.23.106.14) - Compactor for vbucket `default/355` (pid [{type,vbucket},

                                          {name,<<"default/355">>},

                                          {important,false},

                                          {fa,

                                           {#Fun<compaction_new_daemon.13.106246998>,

                                            [<<"default">>,

                                             {355,<<"default/355">>},

                                             {config,

                                              {30,undefined},

                                              {30,undefined},

                                              undefined,false,false,

                                              {daemon_config,30,131072,

                                               20971520}},

                                             false,

                                             {1519515817,0,false}]}}]) terminated unexpectedly (ignoring this): badarg

2018-02-27T15:53:26.873-08:00, compaction_new_daemon:0:warning:message(ns_1@172.23.106.14) - Compactor for vbucket `default/359` (pid [{type,vbucket},

                                          {name,<<"default/359">>},

                                          {important,false},

                                          {fa,

                                           {#Fun<compaction_new_daemon.13.106246998>,

                                            [<<"default">>,

                                             {359,<<"default/359">>},

                                             {config,

                                              {30,undefined},

                                              {30,undefined},

                                              undefined,false,false,

                                              {daemon_config,30,131072,

                                               20971520}},

                                             false,

                                             {1519515817,0,false}]}}]) terminated unexpectedly (ignoring this): badarg

2018-02-27T15:53:46.680-08:00, compaction_new_daemon:0:warning:message(ns_1@172.23.105.62) - Compactor for vbucket `default/343` (pid [{type,vbucket},

                                          {name,<<"default/343">>},

                                          {important,false},

                                          {fa,

                                           {#Fun<compaction_new_daemon.13.106246998>,

                                            [<<"default">>,

                                             {343,<<"default/343">>},

                                             {config,

                                              {30,undefined},

                                              {30,undefined},

                                              undefined,false,false,

                                              {daemon_config,30,131072,

                                               20971520}},

                                             false,

                                             {1519515817,0,false}]}}]) terminated unexpectedly (ignoring this): badarg

2018-02-27T15:53:52.518-08:00, ns_rebalancer:0:info:message(ns_1@172.23.106.14) - Started rebalancing bucket default

2018-02-27T15:53:52.637-08:00, ns_memcached:0:info:message(ns_1@172.23.105.83) - Bucket "default" loaded on node 'ns_1@172.23.105.83' in 0 seconds.

2018-02-27T15:53:52.642-08:00, ns_memcached:0:info:message(ns_1@172.23.105.63) - Bucket "default" loaded on node 'ns_1@172.23.105.63' in 0 seconds.

2018-02-27T15:53:53.751-08:00, ns_vbucket_mover:0:info:message(ns_1@172.23.106.14) - Bucket "default" rebalance does not seem to be swap rebalance

2018-02-27T23:19:44.322-08:00, ns_orchestrator:0:critical:message(ns_1@172.23.106.14) - Rebalance exited with reason {badmatch,

                                 {leader_activities_error,

                                     {default,rebalance},

                                     quorum_lost}}

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

test_1.zip
21.63 MB
28/Feb/18 11:08 PM

Issue Links

is triggered by

MB-28470 CheckpointManager::getNumItemsForCursor is imprecise

Closed

Gerrit Reviews

- Issue Only
- Show All Reviews
- Show Open Reviews
- Show All Issues
- Show Open Issues

No reviews matched the request. Check your Options in the drop-down menu of this sections header.

Activity

People

Assignee:: Arunkumar Senthilnathan (Inactive)

Reporter:: Arunkumar Senthilnathan (Inactive)

Votes:: 0 Vote for this issue

Watchers:: 8 Start watching this issue

Dates

Created:: 28/Feb/18 12:43 PM

Updated:: 15/Mar/18 12:16 PM

Resolved:: 02/Mar/18 9:28 AM

Gerrit Reviews

There are no open Gerrit changes

Show There is 1 closed Gerrit change

Hide There is 1 closed Gerrit change

MB-28431: Avoid getNumItemsForCursor() when getting items to persist: Gerrit Review:

rebalance stuck for more than 8 hours in ubuntu longevity

Details

Description

Attachments

Attachments

Issue Links

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty