Details
-
Bug
-
Resolution: Fixed
-
Test Blocker
-
6.5.0
-
Enterprise Edition 6.5.0 build 3734
-
Untriaged
-
-
Unknown
-
KV-Engine Mad-Hatter Beta
Description
Steps to reproduce
1. Create a 2 node cluster - Both KV nodes
Rebalance Overview
+----------------+----------+--------------+
|
| Nodes | Services | Status |
|
+----------------+----------+--------------+
|
| 172.23.105.220 | [u'kv'] | Cluster node |
|
| 172.23.105.221 | None | <--- IN --- |
|
+----------------+----------+--------------+
|
2. Create a default bucket with replica=1
3. Create 50000 docs with durability=Majority
Bucket statistics
+---------+---------+----------+-----+-------+------------+-----------+-----------+
|
| Bucket | Type | Replicas | TTL | Items | RAM Quota | RAM Used | Disk Used |
|
+---------+---------+----------+-----+-------+------------+-----------+-----------+
|
| default | membase | 1 | 0 | 50000 | 4477419520 | 153456256 | 205281449 |
|
+---------+---------+----------+-----+-------+------------+-----------+-----------+
|
4. Rebalance In 2 new nodes and in parallel start another load of new 50k docs with durability=majority.
Rebalance Overview
+----------------+----------+--------------+
|
| Nodes | Services | Status |
|
+----------------+----------+--------------+
|
| 172.23.105.220 | [u'kv'] | Cluster node |
|
| 172.23.105.221 | [u'kv'] | Cluster node |
|
| 172.23.105.223 | None | <--- IN --- |
|
| 172.23.105.225 | None | <--- IN --- |
|
+----------------+----------+--------------+
|
NOTE: As soon as the rebalance starts we see a lot of com.couchbase.client.core.error.DurabilityImpossibleException for the cruds
5. Rebalance fails
Rebalance exited with reason {mover_crashed,
|
{unexpected_exit,
|
{'EXIT',<0.17469.1>,
|
{{{{{badmatch,
|
[{<0.17464.1>,
|
{done,exit,
|
{socket_closed,
|
{gen_server,call,
|
[<0.16799.1>,
|
{setup_streams,
|
[333,334,335,336,337,338,339,340]},
|
infinity]}},
|
[{gen_server,call,3,
|
[{file,"gen_server.erl"},
|
{line,214}]},
|
{dcp_replicator,
|
'-spawn_and_wait/1-fun-0-',1,
|
[{file,"src/dcp_replicator.erl"},
|
{line,226}]}]}}]},
|
[{misc,
|
sync_shutdown_many_i_am_trapping_exits,
|
1,
|
[{file,"src/misc.erl"},{line,1366}]},
|
{dcp_replicator,spawn_and_wait,1,
|
[{file,"src/dcp_replicator.erl"},
|
{line,248}]},
|
{dcp_replicator,handle_call,3,
|
[{file,"src/dcp_replicator.erl"},
|
{line,121}]},
|
{gen_server,try_handle_call,4,
|
[{file,"gen_server.erl"},{line,636}]},
|
{gen_server,handle_msg,6,
|
[{file,"gen_server.erl"},{line,665}]},
|
{proc_lib,init_p_do_apply,3,
|
[{file,"proc_lib.erl"},{line,247}]}]},
|
{gen_server,call,
|
[<0.16798.1>,
|
{setup_replication,
|
[333,334,335,336,337,338,339,340]},
|
infinity]}},
|
{gen_server,call,
|
['replication_manager-default',
|
{change_vbucket_replication,333,
|
'ns_1@172.23.105.225'},
|
infinity]}},
|
{gen_server,call,
|
[{'janitor_agent-default',
|
'ns_1@172.23.105.220'},
|
{if_rebalance,<0.9696.0>,
|
{update_vbucket_state,321,active,paused,
|
undefined,
|
[['ns_1@172.23.105.220',
|
'ns_1@172.23.105.221']]}},
|
infinity]}}}}}.
|
Rebalance Operation Id = 857ebd24ed77815cb692e2c1c93ae78f
|