[System Test] Rebalance exited with reason {mover_crashed, {unexpected_exit, {memcached_error,internal, "Collections::Manager::update failure: generic failure\"}}

-test tests/eventing/neo/test_eventing_volume.yml -scope tests/eventing/neo/scope_eventing_volume.yml

[2022-06-14T02:59:43-07:00, sequoiatools/couchbase-cli:7.1:bd349b] server-add -c 172.23.104.16:8091 --server-add https://172.23.104.17 -u Administrator -p password --server-add-username Administrator --server-add-password password --services data

[2022-06-14T02:59:56-07:00, sequoiatools/couchbase-cli:7.1:a80939] rebalance -c 172.23.104.16:8091 --server-remove 172.23.104.18 -u Administrator -p password

→  

Error occurred on container - sequoiatools/couchbase-cli:7.1:[rebalance -c 172.23.104.16:8091 --server-remove 172.23.104.18 -u Administrator -p password]

docker logs a80939

docker start a80939

*Unable to display progress bar on this os

JERROR: Rebalance failed. See logs for detailed reason. You can try again.

[user:error,2022-06-14T03:01:20.879-07:00,ns_1@172.23.104.16:<0.9664.0>:ns_orchestrator:log_rebalance_completion:1428]Rebalance exited with reason {mover_crashed,

                              {unexpected_exit,

                               {'EXIT',<0.2418.86>,

                                {{{{badmatch,

                                    {memcached_error,internal,

                                     <<"{\"error\":{\"context\":\"Collections::Manager::update failure: generic failure\"}}">>}},

                                   [{memcached_bucket_config,

                                     ensure_collections,2,

                                     [{file,"src/memcached_bucket_config.erl"},

                                      {line,287}]},

                                    {ns_memcached,

                                     '-perform_very_long_call_with_timing/3-fun-0-',

                                     3,

                                     [{file,"src/ns_memcached.erl"},

                                      {line,905}]},

                                    {ns_memcached,

                                     '-perform_very_long_call/3-fun-0-',2,

                                     [{file,"src/ns_memcached.erl"},

                                      {line,337}]},

                                    {ns_memcached_sockets_pool,

                                     '-executing_on_socket/3-fun-0-',3,

                                     [{file,

                                       "src/ns_memcached_sockets_pool.erl"},

                                      {line,86}]},

                                    {async,'-async_init/4-fun-1-',3,

                                     [{file,"src/async.erl"},{line,191}]}]},

                                  {gen_server,call,

                                   ['ns_memcached-bucket_op',

                                    {get_dcp_docs_estimate,622,

                                     "replication:ns_1@172.23.104.18->ns_1@172.23.104.17:bucket_op"},

                                    300000]}},

                                 {gen_server,call,

                                  [{'janitor_agent-bucket_op',

                                    'ns_1@172.23.104.18'},

                                   {if_rebalance,<0.11843.85>,

                                    {wait_dcp_data_move,

                                     ['ns_1@172.23.104.17',

                                      'ns_1@172.23.104.16'],

                                     622}},

                                   infinity]}}}}}.

Rebalance Operation Id = d3383bad84590d926bdc3a7d5efc2e4c